Mixing Cesium and three.js?

CesiumJS
1

I need to build a UI which lays on top of a Cesium scene and uses three.js to render 3D objects. The two rendering loops will be mutually exclusive – the Cesium rendering loop will be stopped while the three.js UI renders and vice versa. Has anyone tried this before? Any gotchas I should know about upfront?

Thanks,

Peter

2

Hi Peter,

I'm assuming you want the objects on the same 'area' of your page. Not two separate boxes.

I'm investigating the same type of thing, my first guess is it won't work :frowning: We'll have to somehow figure out how to get the render-loops combined on the same OpenGL context, which might be possible, or might be a rewrite of cesium, i'm not quite sure yet.

Cesium's 3D library is pretty whacky for any realtime animation, recreating everything constantly, something simple that would be like 10% of a single core in Three.js pegs my CPU in cesium. There is a lot of VBO creation/destruction going on. Simply rotating the ModelMatrix for objects causes the entire object to be recreated, and there seems to be no realistic object hierarchy, kind of bizarre for a 3D engine.

If we can somehow clone the Cesium GL Context into three.JS's WebGLRenderer (or vice versa) and maintain the state (Scene Projections/etc) I think its our best bet. That could be pretty tricky, and will surely require some internal changes to one of the libraries :frowning:

The other option is just drop back to all custom model rendering and not use three.js (There is a SandCastle example), this obviously is far from ideal.

3

Thanks Phil.

My initial plan was to overlay a different canvas with a transparent background on top of the Cesium canvas (I do simple

overlays now) exclusively for Three.js. So they would be in the same “area” but not in the same container. I don’t need cesium and three.js to be aware of each other. I have my own animation framework controlling both.

My main concern is resource collisions. Since these two are not aware of each other, I wonder if they would mess with each other’s GPU-side state (textures, bump maps, etc?)

I think I’ll try a simple example and see what happens.

Peter

4

Peter - I don’t think you’ll run into problems using separate canvases for Cesium and three.js. Although I’ve only done limited testing, using multiple WebGL contexts on the same page seems to be well supported in modern browsers.

Phil - I’m not aware of any parts of Cesium that rewrite VBOs to change the model-view matrix. Where is that code? I’d like to take a look.

Patrick

5

I concur. They seem to be fine using separate contexts.

Peter

6

Peter -
It's been a long time since I've dealt with 3D hardware so I may be just out of date. I know in windows it used to be forbidden to have two OpenGL contexts on top of each other. Having two contexts on the same page for sure is fine. The resources shouldn't be mixed up, they should be completely separated (I know in NativeGL there is a way to share across contexts/threads, but WebGL is different so i'm unsure).

I'm unsure how you are going to make one 'transparent', but curious if you get this to work. Generally the render target is a separate memory buffer, so having it be 'transparent' doesn't really mean anything. The output is usually a flip or a copy to the target area, flips imply the memory isn't actually moving at all but the HW that updates the RAMDAC has specialized code that says 'hey, at buffer address X actually copy the memory from Y instead'. You would have to render to a target, then copy that target onto the final target. Otherwise you'd just be transparent onto some random garbage or whats already been drawn. Maybe this is implicitly done somehow by WebGL?

Patrick -
The PolyLineCollection will recreate itself on its ModelMatrix change, I think it does this due to 2D/Columbus mode in which it needs to re-project itself. I guess its semi-unavoidable in cases like that. But in the 3D mode there is no need to recreate or project anything.

A bigger issue i'm having right now is needing to update the path, adding a point about every 100ms or so (becoming circular at some threshold). Ideally this would only project only the 'new' point since the others have not changed, but the code is structured such that every point is re-projected, for static views that's fine, but for updating/dynamic content its a bit much. Considering that Cesium has CZML for dynamic/live views I would have thought the base code was a bit more prepared for this.

It performs ok, but its chewing up the entire CPU core (even at a forced 20fps instead of 60fps). And there are other things that on the page that need some of that good old CPU time :slight_smile:

7

Peter - great, thanks for sharing.

Phil - for the use case of adding a new point every 100ms, in the past we’ve created a separate object optimized for this case. It’son the roadmap. We’re currently making some other changes to PolylineCollection for visual quality, which may also help your use case a bit.

Patrick

8

Phil, in response to your concerns about overlapping OpenGL contexts…

I believe that WebGL contexts are actually off-screen buffers that are then composited by the browser. So there shouldn’t be any issues with them overlapping, in the same way that other browser DOM elements are allowed to overlap.

Also, you may be interested to learn that WebGL in both Firefox and Chrome is implemented on top of Direct3D on Windows, not OpenGL. There’s a flag to turn that off, but it is the default in both browsers. They use ANGLE: https://code.google.com/p/angleproject/

Kevin

9

Phil, just to back up what Kevin said: The finished render buffer from OpenGL/Direct3D will still go through normal web compositing with the rest of the page. Low alpha values in the buffer result in the element itself becoming transparent, revealing the web page background, or text, or another , whatever happens to be behind the first canvas. Because of the web browser’s compositing, one render context can appear to be “transparent” on top of another un-related render context, when you use two co-located elements.

10

I am sorry to bump an old thread, but I am also investigating the possibility to re-use the capabilities of Three.js within Cesium.

I have read a thread from Patrick Cozzi about adding Collada support to Cesium: while I am convinced this is necessary, I would think that re-using an already capable framework like Three.js would help a lot. It works, is already feature rich and would enable users to enjoy all the goodies provided out-of-the box: importers, scene graph, animation, effects, etc.

However, being new to the WebGL world, I am not sure of the feasibility of it. My instinct would tell me that two frameworks should be able re-use the same WebGL context within the same canvas but I may be completely wrong.

Any thoughts on that?

Xavier.

11

Hi Xavier,

We’re actually adding support for glTF now, which we’ll convert COLLADA to server-side. More on that later.

I think it is possible to integrate Three.js and Cesium. Cesium would probably control the overall scene rendering, since it does a lot things required to support virtual globes like handling long view distances. We’d add hooks to call into Three.js (or any WebGL engine). I’m not that familiar with Three.js, but I imagine both it and Cesium, of course, would need some modifications, but nothing too big for an initial implementation. We welcome such a contribution.

For more info on Cesium’s rendering, see the Data-Driven Renderer roadmap and Scene.js.

Patrick

12

Rather than reproducing Three.js functionality in Cesium (or vice-versa) why not create a mechanism (or simple example) for synchronizing a Three.js sphere & camera with the Earth and camera in Cesium? Then you could overlay a transparent Three.js canvas on Cesium and use the functionality of both.

If anyone is interested in doing this as a freelance project, we have a project we could possibly use this functionality for. Drop me a line: matt@zebradog.com.

13

I would be interested in to know if anyone has done any work on synchronizing ?

Thanks

14

I am currently trying to synchronize the Three.js coordinate system / camera with the Cesium coordinate system in order to render a Three.js pointcloud with ECEF coordinates on top of the cesium globe.

Can someone give me a hint on how to recreate the Cesium coordinate system in Three.js?

Thanks!

15

Hi,

I have not used Three.js so you might be better off asking their community, but here’s a few ideas:

Let us know how this turns out.

Patrick

16

hi,

has anybody succeeded using three.js together with cesium, in the meantime?

17

Hi,

No one has significantly worked on this yet, but there are some notes and prototypes in #648.

Patrick

18

Hi,

during the work on my thesis I developed a web application using potree, which is based on three.js, together with cesium for rendering.

Some details are described in chapter 6.2 here: https://www.cg.tuwien.ac.at/research/publications/2015/Adorjan-2015/Adorjan-2015-thesis.pdf

On the whole I added an additional render pass to the cesium render loop. In this pass the potree/three.js renderings are done.

19

Matthias,

Great thesis, thanks for sharing! Looks like you understand Cesium’s rendering engine well.

It’s great that you got to work with Michael Wimmer; I really like his work.

If you have the time, we would very much welcome a contribution that allows Cesium to integrate with other engines, and a demo showing how to use it with three.js.

Thanks,

Patrick

20

I like your clear description of ECEF to/from ENU transformations in chapter 5. I am going to save your thesis for reference!

Thanks, Willem