Error: Breadcrumbs 'RHIThread'

Cesium for Unreal
1

Hi,

We’re getting Fatal Error when using Cesium in packaged build. never happens with Cesium disabled.

Never happens in editor, only in packaged game.

It’s happening after a while. Sometimes after 5 minut sometimes after 1hour. After creating automated test that are rotating player camera very quickly and changing pawn locations rapidly we can reproduce it under 3 minutes of use.

It seems that giving Cesium more more memory to operate on helps a bit, but doesn’t eliminate it completely.

Any suggestions on how to fix this?

Best regards,
Damian

[2023.10.11-21.24.03:508][ 78]LogRHI: Error: Breadcrumbs ‘RHIThread’

[2023.10.11-21.24.03:508][ 78]LogWindows: Could not start crash report client using …/…/…/Engine/Binaries/Win64/CrashReportClient-Win64-Debug.exe
[2023.10.11-21.24.03:508][ 78]LogMemory: Platform Memory Stats for Windows
[2023.10.11-21.24.03:508][ 78]LogMemory: Process Physical Memory: 7049.51 MB used, 9041.27 MB peak
[2023.10.11-21.24.03:508][ 78]LogMemory: Process Virtual Memory: 14566.97 MB used, 17170.23 MB peak
[2023.10.11-21.24.03:508][ 78]LogMemory: Physical Memory: 46734.83 MB used, 18502.36 MB free, 65237.20 MB total
[2023.10.11-21.24.03:508][ 78]LogMemory: Virtual Memory: 75828.36 MB used, 12045.05 MB free, 87873.41 MB total
[2023.10.11-21.24.03:508][ 78]SocketIO: SocketIO Invalid appears to have lost connection, reconnecting attempt 4 with delay 25000
[2023.10.11-21.24.03:508][ 78]Message dialog closed, result: Ok, title: The UE-ExteriorRender Game has crashed and will close, text: Fatal error!

[2023.10.11-21.24.03:508][ 78]LogThreadingWindows: Error: Runnable thread RHIThread crashed.
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: === Critical error: ===
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Fatal error!
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000000
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd42b98 ExteriorRender.exe!FD3D12BaseShaderResource::RemoveRenameListener()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd4eab3 ExteriorRender.exe!FD3D12RayTracingGeometry::UnregisterAsRenameListener()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd3548d ExteriorRender.exe!FD3D12CommandContext::RHIBuildAccelerationStructures()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c8d437 ExteriorRender.exe!FRHICommandBuildAccelerationStructures::Execute()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e05f6adb ExteriorRender.exe!FRHICommand<FRHICommandBuildAccelerationStructures,FUnnamedRhiCommand>::ExecuteAndDestruct()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c8e69d ExteriorRender.exe!FRHICommandListBase::Execute()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c7a100 ExteriorRender.exe!operator==()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dbb8cbfe ExteriorRender.exe!TGraphTask<TFunctionGraphTaskImpl<void __cdecl(void),0> >::ExecuteTask()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6de514317 ExteriorRender.exe!FNamedTaskThread::ProcessTasksNamedThread()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6de51482e ExteriorRender.exe!FNamedTaskThread::ProcessTasksUntilQuit()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1dc220c ExteriorRender.exe!FRHIThread::Run()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6deb59282 ExteriorRender.exe!FRunnableThreadWin::Run()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6deb4f350 ExteriorRender.exe!FRunnableThreadWin::GuardedRun()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ffffae0257d KERNEL32.DLL!UnknownFunction
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ffffb6aaa68 ntdll.dll!UnknownFunction
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Crash in runnable thread RHIThread
[2023.10.11-21.24.03:521][ 78]LogExit: Executing StaticShutdownAfterError
[2023.10.11-21.24.03:524][ 78]LogWindows: FPlatformMisc::RequestExit(1)
[2023.10.11-21.24.03:525][ 78]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)
[2023.10.11-21.24.03:525][ 78]LogCore: Engine exit requested (reason: Win RequestExit)

2

Hi @3D_Estate,

Can you help us understand more about what’s going on in your project? In particular,

  • What versions of Unreal Engine and Cesium for Unreal are you using?
  • What are the contents of your level? In particular, how many tilesets are you including? How big are they? The more detailed, the merrier.
  • What are the “Maximum Cached Bytes” values on your tilesets? Does lowering this number make a difference?
3

Hi Janine,

Thank you for quick response.

  • UE 5.2.1, Cesium Plugin 1.30.1
  • Level is a combination of 3d assets and Cesium tiles in the background
  • Maximum Cached Bytes 268435456, (0.25GB) we experimented with this and making it higher seemed to give us less crashes, we changed to 2GB. Will try to lower it.
  • Max Simultaneous Tile Load = 20 - tried to lower to 2, no difference.
  • Loading Descendant Limit 20.
  • We don’t have limit tiles, we can zoom out and see the whole earth, but we’re not doing in during gameplay

image
image907×1474 80.6 KB

image
image907×943 79.1 KB

4

Hi @3D_Estate,

Thank you for sharing those details. I still can’t tell what’s wrong from a glance, so I still have a few more questions.

  • Does this error occur with a packaged level that only contains Cesium World Terrain?
  • Are you able to reproduce this error in Cesium for Unreal Samples?

In the worst case scenario, if we can’t debug it over text, we can take a look at a minimal example level that reproduces this error. But this is reserved as a last resort for our team since it can take us a while to debug individual projects :slight_smile:

5

Hi Janine,

We have spent some time conducting tests as we initially believed that we might be able to address the issue from our end. However, after a week, we decided to disable the plugin in our production builds because the game crashes anywhere between 1 to 180 minutes after starting.

  • I couldn’t reproduce the issue using only the Cesium World Terrain. However, after integrating the sun and sky from the Ultra Dynamic Sky Plugin, the game crashed relatively quickly. Conversely, when I deactivated the Ultra Dynamic Sky and kept everything else intact, the game still crashed.
  • The issue wasn’t replicable on Cesium for Unreal Samples.
  • I observed that by limiting certain variables, the crashes became less frequent but still persisted:
    • cesiumMaxSimultaneousTileLoads: 1
    • cesiumDescendantLimit: 1
    • cesiumEnableFrustumCulling: false
  • I experimented with the value cesiumCachedBytes: 268435456, but it didn’t make a noticeable difference.
  • The crashes also occur when I load tiles and then use debug to suspend loading.
  • Would it be possible for you to examine a small sample of our project? The development build is enought, or would you require access to the Unreal Editor?
  • Additionally, I encountered a different type of error that directly indicates specific lines. This might offer some insights.

I sincerely hope that, together, we can pinpoint and resolve this pesky bug. :wink:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: === Critical error: ===

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: Assertion failed: Index >= ObjFirstGCIndex && Index > ObjLastNonGCIndex [File:D:\build++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectArray.cpp] [Line: 231]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f82d9a7 ExteriorRender.exe!FUObjectArray::AllocateUObjectIndex()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f8286e0 ExteriorRender.exe!UObjectBase::AddObject()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f85f216 ExteriorRender.exe!StaticAllocateObject()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f860a96 ExteriorRender.exe!StaticConstructObject_Internal()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c61271f ExteriorRender.exe!UCesiumGltfComponent::CreateOnGameThread() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\CesiumGltfComponent.cpp:2209]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c63ad00 ExteriorRender.exe!UnrealResourcePreparer::prepareInMainThread() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\Cesium3DTileset.cpp:684]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff745ce39bb ExteriorRender.exe!Cesium3DTilesSelection::TilesetContentManager::finishLoading()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff745c89dc2 ExteriorRender.exe!Cesium3DTilesSelection::Tileset::updateView()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c621fb1 ExteriorRender.exe!ACesium3DTileset::Tick() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\Cesium3DTileset.cpp:1983]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74380316a ExteriorRender.exe!AActor::TickActor()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff7437dc079 ExteriorRender.exe!FActorTickFunction::ExecuteTick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744c3ef4f ExteriorRender.exe!FTickFunctionTask::DoTask()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744c5ef25 ExteriorRender.exe!TGraphTask::ExecuteTask()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73edc4317 ExteriorRender.exe!FNamedTaskThread::ProcessTasksNamedThread()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73edc482e ExteriorRender.exe!FNamedTaskThread::ProcessTasksUntilQuit()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73eddad79 ExteriorRender.exe!FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744cb8120 ExteriorRender.exe!FTickTaskSequencer::ReleaseTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744cc5b59 ExteriorRender.exe!FTickTaskManager::RunTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74427da28 ExteriorRender.exe!UWorld::RunTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744289571 ExteriorRender.exe!UWorld::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff743fd5b3f ExteriorRender.exe!UGameEngine::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd1180f ExteriorRender.exe!FEngineLoop::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2c56c ExteriorRender.exe!GuardedMain()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2c64a ExteriorRender.exe!GuardedMainWrapper()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2f556 ExteriorRender.exe!LaunchWindowsStartup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd42284 ExteriorRender.exe!WinMain()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74695e50a ExteriorRender.exe!__scrt_common_main_seh() [D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ffc20a4257d KERNEL32.DLL!UnknownFunction

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ffc222aaa78 ntdll.dll!UnknownFunction

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:903][ 43]LogExit: Executing StaticShutdownAfterError

[2023.10.24-20.37.29:906][ 43]LogWindows: FPlatformMisc::RequestExit(1)

[2023.10.24-20.37.29:906][ 43]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)

[2023.10.24-20.37.29:906][ 43]LogCore: Engine exit requested (reason: Win RequestExit)

Please let me know if we can do something else to profile the issue.

6

Can you explain in more detail what you mean here?