I’m working on a drone simulation pipeline in NVIDIA Isaac Sim 5.1, using Cesium for Omniverse with Google Photorealistic 3D Tiles as the terrain/environment source. The drone carries a LiDAR (LidarRTX) and three fisheye cameras (front, left, right) feeding an ML training pipeline, so tile resolution directly under and around the drone needs to stay consistently high regardless of what the user is doing elsewhere in the scene.
The issue: since Cesium for Omniverse computes tile selection/LOD independently per active viewport (each viewport contributes its own view state to the tileset’s per-frame update), letting the user freely explore the world through the same camera/viewport used for the drone’s sensors means that navigating far away from the drone evicts or downgrades exactly the tiles the LiDAR/fisheye cameras need, even though the drone itself hasn’t moved.
Our current workaround is a dual-viewport setup: a secondary, low-resolution viewport with a dedicated nadir camera that is continuously synced to the drone’s world position every frame, kept separate from the main viewport the user navigates freely. This keeps the drone-area tiles “claimed” every frame independent of the user’s camera. We’ve also had to disable frustum culling on the tileset in some multi-camera configurations, since with culling enabled tiles visible to secondary cameras but outside the main camera’s frustum were sometimes incorrectly excluded.
The combination is working correctly, but it’s expensive: each viewport is a fully independent render product evaluated every frame, so this roughly doubles baseline RTX rendering cost and doubles the per-frame tile queries against the tileset. Profiling under representative conditions (dual viewport, frustum culling disabled, active precipitation particle system, LiDAR + 3 fisheye streaming) shows RTX rendering alone exceeding 120ms/frame, which is a hard ceiling for our use case.
Before we keep tuning resolution/FOV/clip ranges on the secondary viewport as a stopgap, I wanted to ask the community whether there’s a more native way to solve this:
Is there any mechanism to keep a specific region’s tiles loaded/cached (e.g. anchored to a point or a non-rendering “probe” view) without needing a full active viewport pointed at it?
For multi-camera rigs like this (LiDAR + multiple fisheye + exploration camera), is disabling frustum culling globally really the recommended approach, or is there a per-camera/per-viewport way to guarantee consistent geometry without paying the full cost of unculled rendering everywhere?
Is per-viewport maximumScreenSpaceError (or duplicate tileset instances with different SSE, scoped to different viewports) a supported/recommended pattern for decoupling “cheap exploration view” from “high-fidelity sensor view”, or are there known pitfalls with loading the same ion asset twice this way?
Right now a secondary viewport is required. Ideally we should support virtual cameras without a backing viewport like FCesiumCamera in Cesium for Unreal. This would likely mean creating a CesiumCamera USD prim.
Strange, this shouldn’t be happening. Do you have any screenshots or videos? Ideally you wouldn’t need to disable frustum culling.
Is per-viewport maximumScreenSpaceError (or duplicate tileset instances with different SSE, scoped to different viewports) a supported/recommended pattern for decoupling “cheap exploration view” from “high-fidelity sensor view”, or are there known pitfalls with loading the same ion asset twice this way?
I like the idea of per-viewport maximumScreenSpaceError. I’ll open an issue in Cesium Native for this. The workaround of loading the same tileset twice with different SSE might result in a bunch of z-fighting.
Hi Sean, thanks for the reply and the two issues opened. I think that the virtual camera support will help me a lot in order to get better performance.
In relation with the frustum part, as you can see in the image below, we open 2 cameras (one fisheye and one normal rgb), and the user have zoom into a cluster of solar panels, while the drone’s cameras are watching something else. This does, specially in the fisheye, that appears unloaded tiles in the tileset, which is a problem when our ML model is running.
The best way to mitigate this, without the use of the double viewport setup was to disable frustum, in order to load all the tiles in the tileset, as we can see in the second picture. Luckily for us, disabling frustum doesn’t impact too much our performance, but it’s a good to be able to disable frustum. Probably we can also update the values from clipping in the cameras and achieve the same result that we get disabling the frustum.
Also, a final question. We are currently working of v0.27.0. Those issues open will be apply in the v0.29.0 or they will be apply also to previous releases? Also, we are currently working on Isaac Sim 5.1, there’s any problem if we use v.0.28.0 or we need to update our kit in order to use it?
Just to confirm, when you see the missing tiles, is that when there’s a single active viewport? Or does the problem also occur when there are two? It’s hard to tell from the screenshot if there’s a second viewport.
Also, a final question. We are currently working of v0.27.0. Those issues open will be apply in the v0.29.0 or they will be apply also to previous releases? Also, we are currently working on Isaac Sim 5.1, there’s any problem if we use v.0.28.0 or we need to update our kit in order to use it?
As of now we only support a single version of Kit at a time, so you would need to update IsaacSim to get the fixes.
Hey Sean.
The missing tiles happens when there are two viewports. I show you an example with 2 photos. In the first one, our current setup of 2 viewports, one for the user and the other the camera nadir under the drone. In the second, the RVIZ2 setup, with one fisheye active. As you can see, the border tiles of the fisheye are missing. Only when disabling the frustum those missing tiles appears, due to loading all the tileset.
