Custom image filters in SingleTileImageryProvider

CesiumJS
1

Hi awesome Cesium developers,

I've been using SingleTileImageryProvider to overlay a PNG raster image on top of Cesium base layers, adjust transparency, etc. Very happy with this.

I'd like to add customized image processing filters: I was very impressed to see hue, saturation, brightness, etc., already supported out of the box, but I'd like to do per-pixel custom image processing, in the vein of this tutorial from HTML5Rocks [1]:

[1] http://www.html5rocks.com/en/tutorials/canvas/imagefilters/

I see that SingleTileImageryProvider's (and all the imagery provider classes that use it) requestImage function can yield a canvas or an "image" [2]. I.e., in SingleTileImageryProvider, requestImage gives the "_image" member while in TileCoordinatesImageryProvider, this function returns a 256x256 canvas [3].

[2] https://cesiumjs.org/Cesium/Build/Documentation/SingleTileImageryProvider.html#requestImage
[3] https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/Scene/TileCoordinatesImageryProvider.js#L233

I'm trying to understand what are my choices in inserting custom image processing steps between the imagery provider's original rasters, through a potential canvas that Cesium might be using under the hood, to final display in WebGL. In [4], Scott Hunter proposed a custom imagery provider class that produced custom canvases in its requestImage.

[4] https://groups.google.com/d/msg/cesium-dev/rh48mTIcBe8/I5YMpxyLseEJ

More recently, Kevin Ring in [5] suggests this is could be done as easily as a JS object literal, in a thread that started out discussing the fact that SingleTileImageryProvider could only be constructed with a URL, and not a canvas object.

[5] https://groups.google.com/d/msg/cesium-dev/EBY4pypCRH8/yYzcriiHCAAJ

Instead of diving into creating a custom imagery provider, I noticed that, in an undated set of slides [6], slide 15 ("page" 17), Patrick Cozzi gives "image filters" as an example of contributions from users, which makes me ask if there's something easier I could be looking at to achieve my goal of custom image processing.

[6] https://cesiumjs.org/presentations/Cesium3DMapsOnTheWeb.pdf

Any tips would be most appreciated, thank you!

Ahmed

2

Hi Ahmed,

We have some code in TerriaJS to do this:

https://github.com/TerriaJS/terriajs/blob/master/lib/Map/ImageryProviderHooks.js

This is part of our “region mapping” system, where we encode the ID of rasterized region polygons as colors, and then recolor the polygons on the client. So you probably won’t want to use this directly. But it shows how you can “hook” an arbitrary imagery provider, access its raw image data, and recolor / change it on the fly.

There are fancier things you can do, like do your image processing on the GPU in a shader, but that gets more complicated and may require changes to Cesium itself. The approach here can be done externally and is good enough as long as your image processing function is fast.

Kevin

3

We have some code in TerriaJS to do this:
https://github.com/TerriaJS/terriajs/blob/master/lib/Map/ImageryProviderHooks.js

Very cool, thank you Kevin! I admit to replacing the elegant require()s with replacing them with globals (i.e., `var defined = Cesium.defined;`), but it works well—and I'm especially glad to see that once the recolor function is run, it doesn't need to be re-run whenever I zoom or pan.

I adapted the Imagery Layers Manipulation Sandcastle demo to incorporate ImageryProviderHooks: one can paste the code from here Imagery Layers Manipulation demo for Cesium.js using custom image processing: try at http://cesiumjs.org/Cesium/Apps/Sandcastle/index.html?src=Imagery%20Layers%20Manipulation.html · GitHub into the JavaScript section at http://cesiumjs.org/Cesium/Apps/Sandcastle/index.html?src=Imagery%20Layers%20Manipulation.html and re-run to see the Cesium logo recolored using this functionality.

What would be your recommendation if I wanted to update the recolor function based on user input? I.e., supposing the user can select min/max thresholds using HTML sliders and I wanted to replace the recolorFunc and rerun it. Would I have to somehow trigger another requestImage(), so a new recolorFunc is applied? Is there a way to avoid a web request?

Alternatively, I note that when I set the layer's alpha or saturation, the raster is updated. Looking at the sources, it looks like this update happens in GLSL, and that Dojo is responsible for watching for changes in the layer object's properties. Is it at all possible to dip into that framework to run this kind of recoloring?

Again, many thanks! Best,

Ahmed

4

Replying to my own question: using the insight, about Cesium.ImageryState, at [1], I can do something like this, assuming you have the imagery provider in a `provider` variable, and used my version of ImageryPRoviderHooks.js from the gist [2] in the previous email which stores the recoloring function in the imagery provider as a property, as well as the usual Cesium `viewer` object:

  provider.recolorFunc = (r,g,b,a) => [r, 0, 0, a]; // Just for example!
  var layer = viewer.imageryLayers.get(1);
  layer._imageryCache['[0,0,0]'].state = Cesium.ImageryState.UNLOADED;

Sorry, very hacky I know, mucking around private variables like this, but after running these three lines, my raster disappears; then when I zoom in and out a bit, it reappears with the new recolorFunc applied: :smiley_cat:!

My questions: is there a clean(er) way to do this?

Also, how to avoid the delay before the new recolor function is applied? In [1], from 2012, Scott Hunter mentions an alternative technique: overwrite a now-nonexistent `doneLoading` flag on each tile. Is there something similar I could be doing now?

Finally, how hard would it be to push this to GLSL, the way the built-in properties like hue/saturation/brightness trigger updates?

Thanks for listening!

Ahmed

[1] https://groups.google.com/d/msg/cesium-dev/D9FiNdkbvCQ/5kno9NIIf4wJ
[2] https://gist.github.com/fasiha/63d2dbb36fc88f72c078

5

I hate to revive such an old thread but as someone who is attempting the same (recoloring of imagery layers), I was curious if you found a final solution. I, too, would like to use GLSL to perform the recolor on the GPU. [1] looks promising but uses terrain/DEM data.

[1] https://github.com/PropellerAero/cesium-elevation-gradient

6

Hi Ryan,

There’s no update here, but it would be reasonably straightforward to allow the globe object to take a snippet of GLSL (and some input uniforms) that could be appended to the globe’s fragment shader for post processing. This would be a very welcomed contribution if you are up for it.

To start, search the Cesium code base for:

This could also be achieved with a complete post-processing framework, which we plan to add to Cesium, but the date is TBA so I do not suggest waiting for it.

Patrick