Tile Rendering and Seamless Flying in Multiplayer Environment

Cesium for Unreal
1

We are working on improving our rendering of Cesium Tiles for Quest 2 and Quest 3 devices, but we’re encountering a couple of issues.

  1. LOD Management: When a tile goes out of the camera view, it unloads the higher LOD, making it appear less detailed. Ideally, it should hide the tile (Frustum Culling) and reappear when the tile is back in view. This issue is specific to Quest devices; on Windows, tiles do not unload unnecessarily. Can anyone help us locate this code or understand why tiles revert to the lowest LOD when out of view?
  2. Far Distance Culling: We want to implement FarDistanceCulling for the mesh to control how far from the camera position the mesh should render. When close to the ground, only nearby blocks are visible, so loading distant tiles is unnecessary. Ideally, we would like to cull them based on a lower MaxDrawDistance (similar to StaticMeshComponent behavior). As altitude increases, the draw distance should increase, showing more tiles but at a lower resolution to balance tile loading and quality.

Additionally, we are working on implementing a seamless flying mechanism for multiplayer environments, where users can fly from one part of the globe to another without interruption. Here are the issues we’re currently facing:

  1. Pawn Movement: The pawn moves very far from the origin, causing instability. Should we decrease the globe size as we fly up? How would this work in a multiplayer setting where players are on different parts of the globe? If we reduce the globe size, the mesh size decreases, but the simulated pawn remains the same size, breaking the effect.
  2. Origin Shift Component: We attempted to use the origin shift component, but it didn’t work as expected and caused the pawn to jump back and forth. In World Settings, “World Check Bounds” is set to false.

If anyone has pointers on how to achieve these effects seamlessly for all players in a multiplayer environment, we would greatly appreciate it.

2

Hi @carlrealvr , sounds like you’re doing worthwhile work. I’ll see if I can help out…

When a tile goes out of view, there’s really no guarantee it’s going to stay loaded. It’s not visible after all. In Cesium3dTileset, there’s a Maximum Cache Bytes property that helps control this. The larger it is, the more memory you use, but the more chance that the higher LOD will stick around when it’s not in use

Is this solely based on trying to improve performance? Cesium-native doesn’t do this, but it has been considered. Typically tiles that are far away occupy very few pixels, thus need a low level of detail, and don’t really impact performance as much as you might think.

What kind of instability are you seeing? Do you see you pawn jitter in some way?

3

When a tile goes out of view, there’s really no guarantee it’s going to stay loaded. It’s not visible after all. In Cesium3dTileset, there’s a Maximum Cache Bytes property that helps control this. The larger it is, the more memory you use, but the more chance that the higher LOD will stick around when it’s not in use

I want to control whether a tile will be loaded or unloaded. We don’t want to unload the tile; it can be culled to save rendering resources, but it should not be unloaded. It might not even need to be culled. This decision will depend on the performance we achieve, as we are targeting Quest2 devices with limited resources.

I aim to create an experience where a user sees a city loaded, and as they pan or move their head, they should not experience the loading and unloading of tiles if they are standing still and just moving their head.

To achieve this, I am considering clamping the maximum and minimum LOD (Level of Detail) for tiles within a spherical area.

Min LOD: The minimum LOD or highest quality for tiles. Currently, quality is distance-based, with closer meshes being of higher quality. However, I want to clamp that quality because, on mobile devices, the highest quality might not be necessary.

Distance: This defines how far meshes will be loaded from the camera. Beyond this distance, I don’t want anything to render. For now, this means not loading the minimum LOD. I am using FarCullDistance from Unreal Engine to cull everything beyond this distance, saving some draw calls and primitives from rendering.

Max LOD: This will be the LOD used for tiles within the defined distance. For example, inside the radius, all tiles should load from Min to Max LOD.

This approach will help maintain a consistent visual experience where distant objects are still visible but at a LOD that I determine to be best. This is because the Screen Space Error (SSE) is degrading the quality of meshes too much at a distance, and when I decrease SSE, the highest quality is too demanding for the device.

FirstConcept
FirstConcept696×564 6.31 KB

Is this solely based on trying to improve performance? Cesium-native doesn’t do this, but it has been considered. Typically tiles that are far away occupy very few pixels, thus need a low level of detail, and don’t really impact performance as much as you might think.

Far Culling Distance might not save much, but on mobile devices, it does provide some benefits, so it would be helpful to have this feature.

Currently, I am using Unreal Engine’s Far Culling Distance. The only problem I have is calculating the height. I use the camera’s world position, convert it to longitude, latitude, and height, and then use that height for the Far Culling Distance.

I have another question related to fog culling. I can’t tell if it’s working. How can I debug this?

Additionally, how can I stop tiles from unloading when they are out of view? I’m checking performance and the overall feel of moving in the world.

What kind of instability are you seeing? Do you see you pawn jitter in some way?

Issues with the pawn position moving away from the origin of the world:

  1. Gravity does not work correctly because Unreal uses Z-Gravity, so after a certain distance, things look weird. We allow people to fly around the whole world.
  2. The Niagara component stops working correctly as the world coordinates become very large (this can be fixed using relative space, maybe).
  3. Similar issues occur with some calculations around IK (this can also be fixed using relative space, maybe).

When I use Origin Shift, it makes the pawn jitter, and I don’t know why this is happening.

Ideally, when flying out of a certain radius from the origin, everything should shift towards the origin. By “everything,” I mean:

  1. The current pawn position should become the new GeoCentre, bringing the pawn closer to the origin.
  2. Simulated proxies should also shift by the same distance so that they stay at the same position (longitude and latitude) as they are on the server.

More questions:

We would like to apply the custom implementation ourselves. Please let us know where to find the code for the following:

  1. Deciding the quality of each tile. We understand you are using SSE, and we will make the necessary changes ourselves.
  2. Describing the flow to run cesium-native with the Cesium Unreal Plugin so we can debug everything in real-time. Currently, we need to build libraries each time we make any changes to the cesium-native code.

If possible, we would like to do this without CesiumUnreal, but still be able to see the results, debug the code using breakpoints, and visualize the tiles.

4

@Kevin_Ring Would you be able to answer the above followup questions? Thanks!

5

@Kevin_Ring @Brian_Langevin we’re waiting on a follow up for 9 days and this is pretty crucial for our work. Any help you can offer here?

6

Have you tried turning off “Enable Frustum Culling” for the Cesium3DTileset?

In theory, this shouldn’t unload the tile just because it’s out of the camera’s field of view.

7

Hi @carlrealvr,

I’m afraid your questions are quite difficult, and some of your requests are essentially asking for the plugin to work in a completely different, and, in my opinion, less generally useful or perhaps outright impossible way.

For example, never unloading data once it’s loaded is simply not going to work on a Quest device for any moderately detailed tileset. A detailed model of even a small area of a city is going to be many gigabytes in size and the device will run out of memory.

Brian’s suggestion of disabling frustum culling may help, because it means tiles behind you will still be considered visible for loading/unloading purposes (note that Unreal still does its own frustum culling; these won’t actually be rendered). It will definitely come at a performance and memory cost, though.

As far as understanding how the plugin works, I suggest you use the Developer Setup Instructions to build the plugin yourself, if you’re not already. That will require you to build Cesium Native, which contains a lot of the core selection and loading algorithms. If you build that in the Debug configuration, you should be able to step through there, too.

We’re happy to answer specific questions from there, but I don’t think I can describe to you how the whole thing works start-to-finish in a forum post.

8

Yes we did and it’s making the tiles stay loaded, but the LODs are still changing when out-of-view LODs go to lower quality, and then it seems to look flat when coming back into view.

How can we make the tiles stay active and also not change the LOD?

It’s going to cost more, but we are thinking to clamp the lod so that it does not go to highest quality.

9

Hi @Kevin_Ring - Could you direct us to the section of the code or the process that determines LOD selection based on the distance from the camera or pawn? We’re prepared to make custom modifications to Cesium Native and need guidance on LOD selection. Specifically, we want to understand where and how LODs are chosen for each tile. Our goal is to fix a specific LOD across all tiles, as we find the highest LOD too resource-intensive and the lowest LOD too low-quality at this time, over Quest 2/3. We know this is feasible, as we’ve achieved it with static meshes.

In the documentation it seems like one of the modules, Cesium3DTileSelection is supposed to select the LOD for each tile but it’s not clear how you’d actually select the LODs (ie, how you load a grouping)? Once you point us to the area, we’ll be able to take from there. We’re not asking you to explain, just to point us to the area so we can make some adjustments to fit our unaddressed needs on Quest.

Btw we saw other people asking the same questions, for example here: LOD - Disable Distance Checks + Ensure Highest LOD - #5 by Mark_Grossnickle
@Kevin_Ring it sounds like Unity already does what we’re looking automatically and better than UE - but since we’re built in UE, what can we do to achieve what we’re looking to do? I’m emphasizing that we’re writing this thread over the span of a few weeks - because unfortunately we’re still getting 20 FPS (despite following your optimization tips religiously) which means that our product cannot be shipped unless we try a few more custom changes.

Thanks @Brian_Langevin for helping on this too as this is time sensitive (in case Kevin is gone for the weekend)

10

@carlrealvr

I ended up creating a custom addition to native that allowed me to pass in different camera. Perhaps you could solve your issues by doing the same here since this plugin doesn’t seem to be designed for quick head movement. In my case it ended up being quite simple, I just needed to place the camera above the map and have it pointing downward and all tiles in the area would then load the same LOD and never reload, even if I looked away. If you are moving around it could be more complex but still may be solvable with the custom camera hack.

Side note, I’m not sure why the stance to these requests are “its not an option/not how it works/would break the device” as this is exactly how it works in other plugins like WRLD or MapBox and devs can easily set the LOD to a level that would not break the device if we were given the option.

11

@carlrealvr the LOD selection process starts in Tileset::updateView:

@Mark_Grossnickle I suspect that other products aren’t using a model nearly as detailed as, for example, Google Photorealistic 3D Tiles. But of course it’s always possible I’ve been immersed in this so long I’m just blind to other possibilities. With Cesium for Unreal being open source, I certainly encourage everyone to try out any ideas and report back to the community how it goes!

I think your suggestion of using a stationary-ish top-down camera is a good one. Perhaps it will work for @carlrealvr, too.

12

@Kevin_Ring @Brian_Langevin

  1. we are trying to build cesium-native, using build-helper.sh file which works correctly, but when i try to compile CesiumUnreal plugin i get these errors:

1> Creating library D:\UE\Projects\realvr\Plugins\CesiumForUnreal\Intermediate\Build\Win64\x64\UnrealEditor\Development\CesiumEditor\UnrealEditor-CesiumEditor.sup.lib and object D:\UE\Projects\realvr\Plugins\CesiumForUnreal\Intermediate\Build\Win64\x64\UnrealEditor\Development\CesiumEditor\UnrealEditor-CesiumEditor.sup.exp
1>CesiumIonClient.lib(Connection.obj) : error LNK2019: unresolved external symbol _Thrd_sleep_for referenced in function “void __cdecl std::this_thread::sleep_for<__int64,struct std::ratio<1,1000> >(class std::chrono::duration<__int64,struct std::ratio<1,1000> > const &)” (??$sleep_for@_JU?$ratio@$00$0DOI@@std@@@this_thread@std@@YAXAEBV?$duration@_JU?$ratio@$00$0DOI@@std@@@chrono@1@@Z)
1>D:\UE\Projects\realvr\Plugins\CesiumForUnreal\Binaries\Win64\UnrealEditor-CesiumEditor.dll : fatal error LNK1120: 1 unresolved externals

  1. We are still trying to understand how LOD is being calculated.

There is one function _visitTileIfNeeded which we noticed is deciding if tile is going to load or not, because its doing all the frustum and fog cull checks, and there is this function which called _meetsSse which is calculating the computeScreenSpaceError.

Question: Does ScreenSpaceError determine which LOD to use? From my understanding, LOD selection is based on comparing geometrical error with distance. Is this correct? If the current LOD meets the ScreenSpaceError threshold, it’s acceptable; if not, a higher quality LOD with the next lower GeometricalError is loaded. Please correct me if I’m mistaken, and any guidance on this topic would be greatly appreciated.

  1. How do you test all of your changes , Does Cesium_Trace works for logging ?
  1. We’d still like to be able to run in VS and UE Editor and set breakpoints in cesium native, is there any documentation on how to do this?

Also thanks @Mark_Grossnickle we’re working on trying this idea out

13

we are trying to build cesium-native, using build-helper.sh file which works correctly, but when i try to compile CesiumUnreal plugin i get these errors:

I’m not sure what might cause that error. One guess is that you’re using a newer version of Visual Studio to compile cesium-native, and an older one to compile Cesium for Unreal. Is that a possibility?

Does ScreenSpaceError determine which LOD to use?

Yes. Your explanation is correct. There’s a high-level overview of how it works in this old “Rendering the Whole Wide World on the World Wide Web” presentation (especially starting on slide 15):

  1. How do you test all of your changes , Does Cesium_Trace works for logging ?

For testing, you can write tests, or try it out in Unreal or Unity, or use the debugger to check things. If you want to add quick print statements for debugging, you can use spdlog. Something like this:

SPDLOG_WARN("some debug statement with the value {}.", value);

That will go to the console in tests, or to the output log in Unreal and Unity.

  1. We’d still like to be able to run in VS and UE Editor and set breakpoints in cesium native, is there any documentation on how to do this?

We have documentation about debugging here:

14

CesiumIonClient.lib(Connection.obj) : error LNK2019: unresolved external symbol _Thrd_sleep_for referenced in function “void __cdecl std::this_thread::sleep_for<__int64,struct std::ratio<1,1000> >(class std::chrono::duration<__int64,struct std::ratio<1,1000> > const &)” (??$sleep_for@_JU?$ratio@$00$0DOI@@std@@@this_thread@std@@YAXAEBV?$duration@_JU?$ratio@$00$0DOI@@std@@@chrono@1@@Z)

I just happened to run into this myself. In my case, it was caused by having the “MSVC v143 - VS2022 C++ x64/x86 build tools (v14.34-17.4)” component of Visual Studio 2022 installed. When this particular version of MSVC is installed, Unreal will use it. When you build cesium-native, though, cmake will usually pick the latest version of MSVC available. That meant I was building Cesium Native with MSVC v14.40, and attempting to link with v14.34, which is not a valid thing to do.

You can fix this by either uninstalling the extraneous version of MSVC, or by telling cmake to use it. You can do the latter by adding -DCMAKE_GENERATOR_TOOLSET="version=14.34" to the cmake configure command line when building Cesium Native.

15

@Kevin_Ring @Brian_Langevin

Thank you for the information regarding MSVC; that resolved our issue.

We have another question: Each tile checks if it meets the Screen Space Error (SSE). If it does, it then checks its child tiles. If it does not, it adds the tile to the Load Queue for refinement. However, we are unclear on one part:

How does the system determine whether to load a higher LOD or revert to a lower LOD to meet the SSE? How is this checked and where in the code is this controlled?

Specifically, we want to control this behavior. We examined the ComputeScreenSpaceError() reference to see if it was used elsewhere in a way that might clarify this process.

We tried to customize ComputeScreenSpaceError() based on distance, aiming to clamp the LOD at a certain distance. Our understanding is that if a tile does not want to refine (does not meet SSE), the ratio of Geometrical Error to Distance should meet the SSE criteria, preventing loading more detailed tiles beyond a certain LOD.

Could you provide guidance on how to control this aspect of tile loading and refinement?

Thanks.

16

@Kevin_Ring @Brian_Langevin

We’ve successfully implemented LOD clamping, which is working great and saving at least 150 to 200 draw calls. We’re never loading the highest LOD, and we’ve made it distance-based so we can load the same LOD for a given range. However, I believe this approach isn’t entirely correct, as we still need to calculate the LOD using Screen Space Error (SSE).

I think we should also clamp the minimum LOD, allowing it to interpolate from MaxLOD to MinLOD within a specific range. Within this range, it would use SSE to calculate the LOD. Thank you for the slide you provided; it gave us a better understanding of how everything works.

I have another question regarding occlusion. For some reason, it’s not working for us. When I enable the setting, particularly the “DelayRefinementforOcclusion” option, no tiles load at all. In the logs, I see no occlusion data. Do you have any idea why this might be happening?

@Kevin_Ring @Brian_Langevin but please see above question right before as well. Thanks for taking a look!

17

I have another question regarding occlusion. For some reason, it’s not working for us.

Which version of Cesium for Unreal are you using? We had a bug like this recently, which we fixed in v2.6.0.

18

@Kevin_Ring @Brian_Langevin

We are on v2.6.0.

One more question besides the one about DelayRefinementforOcclusion above, is there any documentation which shows what are the steps to build for Android?

We are tyring to build for Android, we are using Ninja to do so and when we set the toolchain and everything correclty, we get this error:

ninja: error: build.ninja:490: extern/libjpeg-turbo/lib/libturbojpeg.a is defined as an output multiple times

19

@Kevin_Ring @Brian_Langevin

Any idea what could be wrong with building Android above? Should we use something other than ninja?

Thanks.

20

We are on v2.6.0.

Can you please just double check your version? Maybe even try upgrading to v2.7.1? Your symptoms sound identical to the behavior before the recent fix, so let’s check the easy stuff before digging deep.

We are trying to build for Android, we are using Ninja to do so and when we set the toolchain and everything correclty, we get this error

I’ve never seen that error and don’t know what would cause it, but the best source for build instructions is our GitHub Actions setup, which is what builds our official releases. See here for Android on UE5.2:

Or look for Android53 or Android54 to find the build for those versions. For all UE versions, the scripts call into here: