Error: Breadcrumbs 'RHIThread'

Cesium for Unreal
1

Hi,

We’re getting Fatal Error when using Cesium in packaged build. never happens with Cesium disabled.

Never happens in editor, only in packaged game.

It’s happening after a while. Sometimes after 5 minut sometimes after 1hour. After creating automated test that are rotating player camera very quickly and changing pawn locations rapidly we can reproduce it under 3 minutes of use.

It seems that giving Cesium more more memory to operate on helps a bit, but doesn’t eliminate it completely.

Any suggestions on how to fix this?

Best regards,
Damian

[2023.10.11-21.24.03:508][ 78]LogRHI: Error: Breadcrumbs ‘RHIThread’

[2023.10.11-21.24.03:508][ 78]LogWindows: Could not start crash report client using …/…/…/Engine/Binaries/Win64/CrashReportClient-Win64-Debug.exe
[2023.10.11-21.24.03:508][ 78]LogMemory: Platform Memory Stats for Windows
[2023.10.11-21.24.03:508][ 78]LogMemory: Process Physical Memory: 7049.51 MB used, 9041.27 MB peak
[2023.10.11-21.24.03:508][ 78]LogMemory: Process Virtual Memory: 14566.97 MB used, 17170.23 MB peak
[2023.10.11-21.24.03:508][ 78]LogMemory: Physical Memory: 46734.83 MB used, 18502.36 MB free, 65237.20 MB total
[2023.10.11-21.24.03:508][ 78]LogMemory: Virtual Memory: 75828.36 MB used, 12045.05 MB free, 87873.41 MB total
[2023.10.11-21.24.03:508][ 78]SocketIO: SocketIO Invalid appears to have lost connection, reconnecting attempt 4 with delay 25000
[2023.10.11-21.24.03:508][ 78]Message dialog closed, result: Ok, title: The UE-ExteriorRender Game has crashed and will close, text: Fatal error!

[2023.10.11-21.24.03:508][ 78]LogThreadingWindows: Error: Runnable thread RHIThread crashed.
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: === Critical error: ===
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Fatal error!
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000000
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd42b98 ExteriorRender.exe!FD3D12BaseShaderResource::RemoveRenameListener()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd4eab3 ExteriorRender.exe!FD3D12RayTracingGeometry::UnregisterAsRenameListener()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dfd3548d ExteriorRender.exe!FD3D12CommandContext::RHIBuildAccelerationStructures()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c8d437 ExteriorRender.exe!FRHICommandBuildAccelerationStructures::Execute()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e05f6adb ExteriorRender.exe!FRHICommand<FRHICommandBuildAccelerationStructures,FUnnamedRhiCommand>::ExecuteAndDestruct()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c8e69d ExteriorRender.exe!FRHICommandListBase::Execute()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1c7a100 ExteriorRender.exe!operator==()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6dbb8cbfe ExteriorRender.exe!TGraphTask<TFunctionGraphTaskImpl<void __cdecl(void),0> >::ExecuteTask()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6de514317 ExteriorRender.exe!FNamedTaskThread::ProcessTasksNamedThread()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6de51482e ExteriorRender.exe!FNamedTaskThread::ProcessTasksUntilQuit()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6e1dc220c ExteriorRender.exe!FRHIThread::Run()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6deb59282 ExteriorRender.exe!FRunnableThreadWin::Run()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ff6deb4f350 ExteriorRender.exe!FRunnableThreadWin::GuardedRun()
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ffffae0257d KERNEL32.DLL!UnknownFunction
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: [Callstack] 0x00007ffffb6aaa68 ntdll.dll!UnknownFunction
[2023.10.11-21.24.03:508][ 78]LogWindows: Error:
[2023.10.11-21.24.03:508][ 78]LogWindows: Error: Crash in runnable thread RHIThread
[2023.10.11-21.24.03:521][ 78]LogExit: Executing StaticShutdownAfterError
[2023.10.11-21.24.03:524][ 78]LogWindows: FPlatformMisc::RequestExit(1)
[2023.10.11-21.24.03:525][ 78]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)
[2023.10.11-21.24.03:525][ 78]LogCore: Engine exit requested (reason: Win RequestExit)

2

Hi @3D_Estate,

Can you help us understand more about what’s going on in your project? In particular,

  • What versions of Unreal Engine and Cesium for Unreal are you using?
  • What are the contents of your level? In particular, how many tilesets are you including? How big are they? The more detailed, the merrier.
  • What are the “Maximum Cached Bytes” values on your tilesets? Does lowering this number make a difference?
3

Hi Janine,

Thank you for quick response.

  • UE 5.2.1, Cesium Plugin 1.30.1
  • Level is a combination of 3d assets and Cesium tiles in the background
  • Maximum Cached Bytes 268435456, (0.25GB) we experimented with this and making it higher seemed to give us less crashes, we changed to 2GB. Will try to lower it.
  • Max Simultaneous Tile Load = 20 - tried to lower to 2, no difference.
  • Loading Descendant Limit 20.
  • We don’t have limit tiles, we can zoom out and see the whole earth, but we’re not doing in during gameplay

image
image907×1474 80.6 KB

image
image907×943 79.1 KB

4

Hi @3D_Estate,

Thank you for sharing those details. I still can’t tell what’s wrong from a glance, so I still have a few more questions.

  • Does this error occur with a packaged level that only contains Cesium World Terrain?
  • Are you able to reproduce this error in Cesium for Unreal Samples?

In the worst case scenario, if we can’t debug it over text, we can take a look at a minimal example level that reproduces this error. But this is reserved as a last resort for our team since it can take us a while to debug individual projects :slight_smile:

5

Hi Janine,

We have spent some time conducting tests as we initially believed that we might be able to address the issue from our end. However, after a week, we decided to disable the plugin in our production builds because the game crashes anywhere between 1 to 180 minutes after starting.

  • I couldn’t reproduce the issue using only the Cesium World Terrain. However, after integrating the sun and sky from the Ultra Dynamic Sky Plugin, the game crashed relatively quickly. Conversely, when I deactivated the Ultra Dynamic Sky and kept everything else intact, the game still crashed.
  • The issue wasn’t replicable on Cesium for Unreal Samples.
  • I observed that by limiting certain variables, the crashes became less frequent but still persisted:
    • cesiumMaxSimultaneousTileLoads: 1
    • cesiumDescendantLimit: 1
    • cesiumEnableFrustumCulling: false
  • I experimented with the value cesiumCachedBytes: 268435456, but it didn’t make a noticeable difference.
  • The crashes also occur when I load tiles and then use debug to suspend loading.
  • Would it be possible for you to examine a small sample of our project? The development build is enought, or would you require access to the Unreal Editor?
  • Additionally, I encountered a different type of error that directly indicates specific lines. This might offer some insights.

I sincerely hope that, together, we can pinpoint and resolve this pesky bug. :wink:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: === Critical error: ===

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: Assertion failed: Index >= ObjFirstGCIndex && Index > ObjLastNonGCIndex [File:D:\build++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectArray.cpp] [Line: 231]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f82d9a7 ExteriorRender.exe!FUObjectArray::AllocateUObjectIndex()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f8286e0 ExteriorRender.exe!UObjectBase::AddObject()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f85f216 ExteriorRender.exe!StaticAllocateObject()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73f860a96 ExteriorRender.exe!StaticConstructObject_Internal()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c61271f ExteriorRender.exe!UCesiumGltfComponent::CreateOnGameThread() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\CesiumGltfComponent.cpp:2209]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c63ad00 ExteriorRender.exe!UnrealResourcePreparer::prepareInMainThread() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\Cesium3DTileset.cpp:684]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff745ce39bb ExteriorRender.exe!Cesium3DTilesSelection::TilesetContentManager::finishLoading()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff745c89dc2 ExteriorRender.exe!Cesium3DTilesSelection::Tileset::updateView()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73c621fb1 ExteriorRender.exe!ACesium3DTileset::Tick() [C:\WS_3dEstate_DP\SM_NDI_DP\Plugins\CesiumForUnreal\Source\CesiumRuntime\Private\Cesium3DTileset.cpp:1983]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74380316a ExteriorRender.exe!AActor::TickActor()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff7437dc079 ExteriorRender.exe!FActorTickFunction::ExecuteTick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744c3ef4f ExteriorRender.exe!FTickFunctionTask::DoTask()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744c5ef25 ExteriorRender.exe!TGraphTask::ExecuteTask()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73edc4317 ExteriorRender.exe!FNamedTaskThread::ProcessTasksNamedThread()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73edc482e ExteriorRender.exe!FNamedTaskThread::ProcessTasksUntilQuit()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73eddad79 ExteriorRender.exe!FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744cb8120 ExteriorRender.exe!FTickTaskSequencer::ReleaseTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744cc5b59 ExteriorRender.exe!FTickTaskManager::RunTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74427da28 ExteriorRender.exe!UWorld::RunTickGroup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff744289571 ExteriorRender.exe!UWorld::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff743fd5b3f ExteriorRender.exe!UGameEngine::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd1180f ExteriorRender.exe!FEngineLoop::Tick()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2c56c ExteriorRender.exe!GuardedMain()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2c64a ExteriorRender.exe!GuardedMainWrapper()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd2f556 ExteriorRender.exe!LaunchWindowsStartup()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff73bd42284 ExteriorRender.exe!WinMain()

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ff74695e50a ExteriorRender.exe!__scrt_common_main_seh() [D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ffc20a4257d KERNEL32.DLL!UnknownFunction

[2023.10.24-20.37.29:894][ 43]LogWindows: Error: [Callstack] 0x00007ffc222aaa78 ntdll.dll!UnknownFunction

[2023.10.24-20.37.29:894][ 43]LogWindows: Error:

[2023.10.24-20.37.29:903][ 43]LogExit: Executing StaticShutdownAfterError

[2023.10.24-20.37.29:906][ 43]LogWindows: FPlatformMisc::RequestExit(1)

[2023.10.24-20.37.29:906][ 43]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)

[2023.10.24-20.37.29:906][ 43]LogCore: Engine exit requested (reason: Win RequestExit)

Please let me know if we can do something else to profile the issue.

6

Can you explain in more detail what you mean here?

7

I seem to have found a solution to this problem, and I have also been affected by it. This year, I took over a Cesium project from a former programmer. In my development environment, the packaged program always crashes when running, which is consistent with the problem raised by the person. Almost 99% of the camera crashes when quickly rotating. The development environment of other employees in our company is also not good. Every time I package and release, I have to go to the former programmer’s computer to package, which is very troublesome. But if he packages it, there won’t be any problems, the key is the same code. I suspect it may be related to the environment, so I exported his Visual Studio configuration file and reinstalled my Visual Studio (importing his configuration file). Something magical happened, and the program was packaged without any problems. I didn’t continue to investigate which component had the problem. At first, I suspected that the version of the Windows SDK was too low, but it didn’t seem to be. I guess it’s related to some components not being added. I will show the exported configuration file of VS below, hoping to help the next person who encounters this problem.
{
“version”: “1.0”,
“components”: [
“Microsoft.VisualStudio.Component.CoreEditor”,
“Microsoft.VisualStudio.Workload.CoreEditor”,
“Microsoft.Net.Component.4.8.SDK”,
“Microsoft.Net.Component.4.7.2.TargetingPack”,
“Microsoft.Net.ComponentGroup.DevelopmentPrerequisites”,
“Microsoft.VisualStudio.Component.TypeScript.TSServer”,
“Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions”,
“Microsoft.VisualStudio.Component.JavaScript.TypeScript”,
“Microsoft.VisualStudio.Component.Roslyn.Compiler”,
“Microsoft.Component.MSBuild”,
“Microsoft.VisualStudio.Component.Roslyn.LanguageServices”,
“Microsoft.VisualStudio.Component.TextTemplating”,
“Microsoft.VisualStudio.Component.NuGet”,
“Microsoft.VisualStudio.Component.SQL.CLR”,
“Microsoft.Component.ClickOnce”,
“Microsoft.VisualStudio.Component.ManagedDesktop.Core”,
“Microsoft.NetCore.Component.Runtime.9.0”,
“Microsoft.NetCore.Component.Runtime.8.0”,
“Microsoft.NetCore.Component.SDK”,
“Microsoft.VisualStudio.Component.FSharp”,
“Microsoft.ComponentGroup.ClickOnce.Publish”,
“Microsoft.NetCore.Component.DevelopmentTools”,
“Microsoft.VisualStudio.Component.AppInsights.Tools”,
“Microsoft.Net.Component.4.8.TargetingPack”,
“Microsoft.Net.ComponentGroup.4.8.DeveloperTools”,
“Microsoft.VisualStudio.Component.DiagnosticTools”,
“Microsoft.VisualStudio.Component.EntityFramework”,
“Microsoft.VisualStudio.Component.Debugger.JustInTime”,
“Component.Microsoft.VisualStudio.LiveShare.2022”,
“Microsoft.VisualStudio.Component.IntelliCode”,
“Component.VisualStudio.GitHub.Copilot”,
“Microsoft.Net.Component.4.6.2.TargetingPack”,
“Microsoft.Component.PythonTools”,
“Microsoft.VisualStudio.Component.VC.CoreIde”,
“Microsoft.VisualStudio.Component.Windows10SDK”,
“Microsoft.VisualStudio.Component.VC.Tools.x86.x64”,
“Microsoft.VisualStudio.Component.Graphics.Tools”,
“Microsoft.VisualStudio.Component.VC.DiagnosticTools”,
“Microsoft.VisualStudio.Component.Windows11SDK.22621”,
“Microsoft.VisualStudio.Workload.Python”,
“Component.OpenJDK”,
“Microsoft.VisualStudio.ComponentGroup.MSIX.Packaging”,
“Component.Android.SDK.MAUI”,
“Microsoft.VisualStudio.Component.ManagedDesktop.Prerequisites”,
“Microsoft.VisualStudio.Component.DotNetModelBuilder”,
“Microsoft.VisualStudio.ComponentGroup.WindowsAppSDK.Cs”,
“Microsoft.ComponentGroup.Blend”,
“Microsoft.VisualStudio.Workload.ManagedDesktop”,
“Microsoft.VisualStudio.Component.VC.ATL”,
“Microsoft.VisualStudio.Component.VC.Redist.14.Latest”,
“Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core”,
“Microsoft.VisualStudio.Component.Windows11Sdk.WindowsPerformanceToolkit”,
“Microsoft.VisualStudio.Component.CppBuildInsights”,
“Microsoft.VisualStudio.ComponentGroup.WebToolsExtensions.CMake”,
“Microsoft.VisualStudio.Component.VC.CMake.Project”,
“Microsoft.VisualStudio.Component.VC.TestAdapterForBoostTest”,
“Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest”,
“Microsoft.VisualStudio.Component.VC.ASAN”,
“Microsoft.VisualStudio.Component.Vcpkg”,
“Microsoft.Component.VC.Runtime.UCRTSDK”,
“Microsoft.VisualStudio.Component.VC.140”,
“Microsoft.Component.NetFX.Native”,
“Microsoft.VisualStudio.Component.Graphics”,
“Microsoft.VisualStudio.ComponentGroup.UWP.Xamarin”,
“Microsoft.VisualStudio.ComponentGroup.UWP.Support”,
“Microsoft.VisualStudio.Component.VC.Tools.ARM64EC”,
“Microsoft.VisualStudio.Component.UWP.VC.ARM64EC”,
“Microsoft.VisualStudio.Component.VC.Tools.ARM64”,
“Microsoft.VisualStudio.Component.UWP.VC.ARM64”,
“Microsoft.VisualStudio.Component.VC.Tools.ARM”,
“Microsoft.VisualStudio.ComponentGroup.UWP.VC”,
“Microsoft.VisualStudio.Workload.NativeDesktop”,
“Microsoft.VisualStudio.Component.WindowsAppSdkSupport.CSharp”,
“Microsoft.VisualStudio.ComponentGroup.WindowsAppDevelopment.Prerequisites”,
“Microsoft.VisualStudio.ComponentGroup.UWP.NetCoreAndStandard”,
“Microsoft.VisualStudio.Workload.Universal”,
“Component.Android.NDK.R23C”,
“Component.MDD.Android”,
“Microsoft.VisualStudio.Workload.NativeMobile”,
“Microsoft.Net.ComponentGroup.TargetingPacks.Common”,
“Microsoft.VisualStudio.Component.NuGet.BuildTools”,
“Component.Unreal”,
“Microsoft.VisualStudio.Workload.NativeGame”,
“Microsoft.NetCore.Component.Runtime.6.0”,
“Microsoft.VisualStudio.Component.VC.14.29.16.11.x86.x64”,
“Microsoft.VisualStudio.Component.VC.14.29.16.11.ATL”,
“Microsoft.VisualStudio.Component.VC.14.36.17.6.x86.x64”
],
“extensions”:
}

8

Hi @MarsBaron, welcome to the community!

If you can share the call stack of the crash, that might help us diagnose this.

My first guess is that it’s related to the version of the MSVC compiler that is used on the “working” machine versus the others. A really common cause of crashes is to build with a newer MSVC compiler, but not install the MSVCRT redistributable for that version, so your system will attempt to use an older one.

It’s easy to test if this is the problem. On a machine where it crashes, and without changing anything about your build or packaging process, install the vc_redist.x64.exe from this page:

(direct link to the download: https://aka.ms/vs/17/release/vc_redist.x64.exe)

With any luck, that will fix the problem.