I would like to have object (ID) information for every pixel (in order to e.g. pass it or derived information into the PostProcessingStage or associate rendered colors to objects afterwards).
What I have tried
I have found the PickFramebuffer class used in Scene.pick and am considering writing a modified version of it.
I am also considering just calling Scene.pick repeatedly for every pixel (which is of course slow, but it would not have to happen every frame anyway).
Context
I want to make smarter post-processing decisions in the shader or take the rendered color and apply it to the point-cloud (which doesn’t come with color).
I don’t think there’s a built in way to do this. For Scene.pick, wouldn’t you have to call it on every pixel for every frame that the view changes?
If you just wanted to apply some kind of shader/color effect to just the point cloud, it might be a lot more efficient to modify the drawing command for the Primitive with a custom shader that only applies to it. You’d have to potentially modify the source code though. The “graphics tech in Cesium” is a good series on the blog to explain how this is all set up:
You can see everything in the WebGL category here:
Another approach I can think of that might take more work to do but would potentially make it a lot simpler for the user, is we already know something about each pixel, for example we know the depth of it by looking up the depth/z-buffer. You could potentially add vertex attributes to everything, and draw them onto a buffer in the same way, so that what you end up with is a texture that tells you for each pixel, what kind of object it is. But again, this would have to be more of a core change to the rendering I think.
I also just remembered that the post process pipeline has a “per feature” system, so you could potentially apply a post process only to certain primitives (and since 3D tilesets are primitive this could easily solve what you’re attempting to do). Check out this Sandcastle:
Another approach I can think of that might take more work to do but would potentially make it a lot simpler for the user, is we already know
something about each pixel, for example we know the depth of it by looking up the depth/z-buffer. You could potentially add vertex attributes to everything, and draw them onto a buffer in the same way, so that what you end up with is a texture that tells you for each pixel,
what kind of object it is. But again, this would have to be more of a core change to the rendering I think.
So you’re saying that there is no ID buffer rendered, yet?
If I want to transfer color from the Framebuffer to individual points I will need to pick the individual points, though. Some tests I did recently seem to indicate that that is not possible? (I only get the tileset when picking)
Also I would like to pass tileset features (like semantic info about a point like “is this a tree or a car?”) into the shader, not just activate it for a certain tileset.
Actually, I was wrong. There is indeed an ID buffer rendered with a unique color per object, which you could grab into your shader/custom pass to see what objects are at any given pixel.
By default, calling scene.pick will render just a 3x3 texture around the mouse:
You can change this to be the whole screen (so that’d be several orders of magnitude faster than calling pick for every pixel). The way the pick texture works right now is every object is given a unique ID/color. I think what you’re going for is each type of object gets the same color. You can see where the colors are assigned/created here:
I do know that the 3D Tiles styling system does let you color points by classification/any other per-point data. The picture below shows you what this looks like:
Let me know if this is what you’re going for. There’s not a lot of documentation on this, but this thread has some example code of styling a point cloud based on per-point data:
You are correct, you would need to give each point a unique ID and then you’ll be able to pick it. That snippet you linked to creates enough pickIDs for each feature. You can see where it assigns these pickIDs here:
And if there are no features present, I think each tile just gets a unique pickID then.
I am curious if you got a chance to look into the styling language. If the goal is to color points based on some per-point metadata this should be the easiest option. Let me know if that doesn’t work for your use case.
The other approach that could be faster coputationally is instead of worrying about writing to the pick texture, reading back the pixels, and then sending that information to your post process shader, you could just create a custom pass that renders the point cloud with custom colors according to some per-point data. You can then take this texture and pass it to your post process, and if you encounter, say red, you’d do something special for those points in your shader. There was some discussion on rendering to a texture in this thread:
