Cesium for Unreal v2.1.0 - still missing tiles at the edges in VR

Lildreas · December 8, 2023, 3:47pm

Hi,

we are using the Varjo XR-3 headsets for our flight simulator, created with UE 5.3.2 and the latest CESIUM plugin.

We have the issue that at the outmost edges of the view in the headset we see edges of missing tiles, when we move the head to the side.

The tiles should always be there, as we are not using Engine → Rendering Occlusion Culling.

CESIUM experimental Occlusion Culling is deactivated, too.

Our other settings:

Any ideas?

Thanks & best
Andreas

Kevin_Ring · December 11, 2023, 2:26am

It sounds like you could be running into this issue:

github.com/CesiumGS/cesium-unreal

Tiles on edges of wide frustum don't render

opened 09:00PM - 21 Nov 23 UTC

closed 09:28PM - 30 Nov 23 UTC

csciguy8

On Windows 10, Unreal Engine 5.3.1, with a Nvidia Quadro K2200. 1) Load cesiu…m-unreal-samples, level 05_CesiumWorldPostProcessing. 2) Push the play button to go into Play in Editor mode 3) Move the camera side to side Granted, this is an old video card, and Unreal Engine does complain to me on startup (see popup). Still, I wouldn't expect tiles on the edges to be missing simply because of D3D 12 compatibility. Also odd, I don't see this behavior when not in Play in Editor mode. ![TilesOnEdges](https://github.com/CesiumGS/cesium-unreal/assets/130494071/a9beaec0-bd0e-47e6-b8ac-bd64b83cf7e2) ![image](https://github.com/CesiumGS/cesium-unreal/assets/130494071/f0c0978a-342f-42aa-bb26-2cf646473d04)

It was fixed in v2.1.0. Have you upgraded?

If that’s not the problem, the only other thing I can suggest is that you try disabling frustum culling. It shouldn’t be necessary, but if the device is for some reason taking more than one frame to show a previously-invisible tile, those gaps can appear.

Lildreas · December 12, 2023, 10:43am

Hi @Kevin_Ring ,

we have this issue with v2.1.0, too, tested with the latest version.

And there is no “Enable Frustum Culling” option on the CesiumWorldTerrain. I searched for it, nothing.

So, first thing, any idea why this option doesn’t show up? We are using Google 3D Tiles.

And if we find it, maybe the issue is that in VR another frustrum value would be needed? Can this value be changed someplace?

Thanks & best
Andreas

Kevin_Ring · December 12, 2023, 11:14pm

Hmm that is very strange. The Enable Frustum Culling option is found under Cesium → Tile Culling, and indeed it is on my system:

I’m struggling to imagine why it would be missing in yours. Did you install from the Unreal Engine Marketplace? From our GitHub releases page? From a custom build of your own?

Lildreas · December 13, 2023, 10:28am

Hi @Kevin_Ring,

it’s a mystery. I installed UE5.3.2 directly from EPIC, and the v2.1.0 CESIUM plugin from the Marketplace. Just removed it and installed it again, but still no “Enable Frustrum Culling” option.

We are using the Varjo high fidelity VR/MR headsets for the flight sim, and I guess they might change the Frustrum based on their field of view… but in their Varjo plugin I cannot find any settings regarding Frustrum either. And why should this affect the apearance of the option in the CESIUM plugin? I’m stumped.

Best
Andreas

Kevin_Ring · December 18, 2023, 9:06am

Can you try loading the Cesium for Unreal Samples project and see if it’s also missing there?

Lildreas · December 18, 2023, 5:46pm

Hi @Kevin_Ring,

found it! I used the Samples project as suggested, and the “Enable Frustrum Culling” checkbox was there.

Then I installed the VarjoOpenXR plugin, but it still was there, so I removed it again to not make the problem any more complex.

Afterwards I started to check and uncheck every single option in the CESIUM World Terrain that I use, and I watched “Enable Frustrum Culling”…

…and it disappeared, after I checked “Use LOD Transitions”…

…the “Enable Frustrum Culling” option was gone:

This happened regardless of the WorldTerrain used, I tested both with CESIUM and Google 3DTiles.

Hope this helps to solve the problem,
best
Andreas

Kevin_Ring · December 19, 2023, 5:43am

Ah ok, that makes sense. I believe frustum culling is forced off when LOD transitions are enabled. But it’s worth a try, with LOD transitions off and also frustum culling off, to see if it fixes your original problem.

Beyond that, I think we’ll need your help to reproduce the problem so that we can debug it. We don’t have a Varjo XR-3, but perhaps you can find a way to reproduce the problem without the device?

Lildreas · December 19, 2023, 2:35pm

Hi @Kevin_Ring,
so I switched LOD transitions OFF, and Frustrum Culling OFF, too.
And I got the exact same result, when rotating the head I saw the edges of Tiles disappearing.
Then I switched Frustrum Culling ON again, nothing changed, the Tiles disappeared the same.
Seams like two issues:

Frustrum not wide enough, so culling at the edges is visible
Switching Frustrum Culling ON/OFF doesn’t change anything, behaves as if always ON

Kevin_Ring · December 20, 2023, 11:13am

Seams like two issues

Ok, but those two issues don’t show up on other devices, as far as I know. So there’s likely something more subtle going on. And I don’t see how we can debug it without being able to reproduce it first. Can you help us do that?

Lildreas · December 21, 2023, 4:30pm

Hi @Kevin_Ring,

of course I can help. I already started with the CESIUM sample project with the default map and installed the VarjoOpenXR plugin, I didn’t change any settings.

With this I got a working frustrum setting in the VR headset view, if Frustrum Culling was OFF there was no visible culling on the edges, if Frustrum Culling was ON the tiles at the edges visibly disappeared.

So, first problem, first finding:

“Frustrum Culling == ON” causes visible culling artefacts at the frustrum edges for Varjo VR/MR

Can there be done something about this? Can we change the Frustrum Culling dimensions somewhere at runtime, or in the CESIUM plugin code, or in the Engine Source? To fix some issue with the Frustrum you did some changes, so I guess there might be something in the CESIUM plugin.

Second Problem:

In my project switching Frustrum Culling ON/OFF does not change anything, it always behaves as if Frustrum Culling is ON and there are tiles missing at the edges, even if the Culling is set to OFF.

For this I started to change the settings from the CESIUM Samples default map to my project settings for the flight simulator, and after every change I tested again with the Frustrum Culling for being active even if switched OFF in the setting.

So far, I tested:

DirectX 11 SM5 → DirectX 12 SM6 => All well, no artifacts with Frustrum Culling == OFF
Switch to Forward Shading => All well, no artifacts with Frustrum Culling == OFF
Switched ON “Instanced Stereo” rendering => All well, no artifacts with Frustrum Culling == OFF

These are the most important settings, and so far no luck, in the demo project the Frustrum Culling == OFF setting is still working.

I’ll have to continue after the Christmas period.

So, BTW, have a nice Christmas time!

Best
Andreas

P.S.:
If I can help in any other way with finding the issues, just tell me.

Kevin_Ring · December 22, 2023, 1:25pm

Thanks @Lildreas. By any chance are you able to reproduce the problem outside of the Varjo? Perhaps by using a different/unusual aspect ratio for your window?

The code that computes the view parameters is here:

github.com

CesiumGS/cesium-unreal/blob/01cb4e851fa044ec1269b9e8457c6a2d6f045a67/Source/CesiumRuntime/Private/Cesium3DTileset.cpp#L1326


      
                  pCameraManager->GetCameras();
              cameras.reserve(cameras.size() + extraCameras.Num());
              for (auto cameraIt : extraCameras) {
                cameras.push_back(cameraIt.Value);
              }
            }
          
            return cameras;
          }
          
          std::vector<FCesiumCamera> ACesium3DTileset::GetPlayerCameras() const {
            UWorld* pWorld = this->GetWorld();
            if (!pWorld) {
              return {};
            }
          
            double worldToMeters = 100.0;
            AWorldSettings* pWorldSettings = pWorld->GetWorldSettings();
            if (pWorldSettings) {
              worldToMeters = pWorldSettings->WorldToMeters;
            }

You could try stepping through there in the debugger on the device and see if anything jumps out at you as wrong.

There’s no direct way to change the frustum size. But the CesiumCameraManager Blueprint API can be used to provide custom camera parameters (including field-of-view). So you could write a Blueprint that provides Cesium with a widened one. This post is the best documentation we have at the moment for using this API. It’s solving a different problem, but hopefully will give you a sense of how it works:

Merry Christmas to you as well!

Brian_Langevin · January 3, 2024, 5:56pm

Hi @Lildreas ,

I’d also be interested to see what your camera settings are for you Dynamic Pawn? (example below)

From the bug that Kevin linked, we found that our code did not support AspectRatio_MaintainYFOV, but there may be other combinations of features that can cause problems. Basically, cesium native’s tile selection logic was assuming a frustum that wasn’t as wide as what Unreal was actually using. Thus no tiles on the edges of the screen.