Cesium crashing Linux Multiplayer Server

Cesium for Unreal
1

I am using Cesium in a multiplayer map that uses Cesium and it crashes with this stack. Any ideas why this may happen or how to fix it?

Thank you,

-Brandon

[2025.02.28-03.57.38:494][401]LogLoad: Took 1.142653 seconds to LoadMap(/RedRocks2025/Levels/RedRocks_Primary)
[2025.02.28-03.57.38:494][401]LogGlobalStatus: LoadMap Load map complete /RedRocks2025/Levels/RedRocks_Primary
[2025.02.28-03.57.38:615][404]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to read memory at address 0x0000000000010008

[2025.02.28-03.57.38:615][404]LogCore: Fatal error!

0x00000000106a6833 VirtuosoServer!rapidjson::GenericValue<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >::SetStringRaw(rapidjson::GenericStringRef, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator&)(+0x63)
0x00000000106a6125 VirtuosoServer!rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>::String(char const*, unsigned int, bool)()
0x000000001077ef26 VirtuosoServer!void rapidjson::GenericReader<rapidjson::UTF8, rapidjson::UTF8, rapidjson::CrtAllocator>::ParseString<0u, rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>&, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>&, bool)()
0x000000001077f158 VirtuosoServer!void rapidjson::GenericReader<rapidjson::UTF8, rapidjson::UTF8, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>&, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>&)()
0x000000001077f158 VirtuosoServer!void rapidjson::GenericReader<rapidjson::UTF8, rapidjson::UTF8, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>&, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>&)()
0x000000001077e8f0 VirtuosoServer!rapidjson::ParseResult rapidjson::GenericReader<rapidjson::UTF8, rapidjson::UTF8, rapidjson::CrtAllocator>::Parse<0u, rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>&, rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>&)()
0x000000001077e7ac VirtuosoServer!rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>& rapidjson::GenericDocument<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, rapidjson::CrtAllocator>::ParseStream<0u, rapidjson::UTF8, rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream> >(rapidjson::EncodedInputStream<rapidjson::UTF8, rapidjson::MemoryStream>&)()
0x00000000107cf996 VirtuosoServer!UnknownFunction()
0x0000000010893c53 VirtuosoServer!CesiumAsync::CesiumImpl::QueuedScheduler::dispatchInternal(bool)()
0x0000000010893b7a VirtuosoServer!CesiumAsync::CesiumImpl::QueuedScheduler::dispatchQueuedContinuations()()
0x00000000107ff26b VirtuosoServer!Cesium3DTilesSelection::Tileset::updateView(std::__1::vector<Cesium3DTilesSelection::ViewState, std::__1::allocatorCesium3DTilesSelection::ViewState > const&, float)()
0x000000000d8690cf VirtuosoServer!ACesium3DTileset::Tick(float) [V:/Virtuoso2023/Virtuoso/Plugins/CesiumForUnreal/Source/CesiumRuntime/Private/Cesium3DTileset.cpp:2032]
0x0000000009e15a3f VirtuosoServer!FActorTickFunction::ExecuteTick(float, ELevelTick, ENamedThreads::Type, TRefCountPtr const&) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/Actor.cpp:238]
0x000000000b19e82a VirtuosoServer!FTickFunctionTask::DoTask(ENamedThreads::Type, TRefCountPtr const&) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/TickTaskManager.cpp:275]
0x000000000b19e279 VirtuosoServer!TGraphTask::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type, bool) [U:/Source/UE5Git/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:1235]
0x0000000005281d66 VirtuosoServer!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:760]
0x00000000052806af VirtuosoServer!FNamedTaskThread::ProcessTasksUntilQuit(int) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:650]
0x000000000527f272 VirtuosoServer!FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete(TArray<TRefCountPtr, TSizedInlineAllocator<4u, 32, TSizedDefaultAllocator<32> > > const&, ENamedThreads::Type) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:2122]
0x000000000b197708 VirtuosoServer!FTickTaskSequencer::ReleaseTickGroup(ETickingGroup, bool) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/TickTaskManager.cpp:555]
0x000000000b1917ba VirtuosoServer!FTickTaskManager::RunTickGroup(ETickingGroup, bool) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/TickTaskManager.cpp:1582]
0x000000000a7fddd2 VirtuosoServer!UWorld::Tick(ELevelTick, float) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/LevelTick.cpp:1610]
0x000000000a5795fe VirtuosoServer!UGameEngine::Tick(float, bool) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Engine/Private/GameEngine.cpp:1782]
0x000000000bc004fb VirtuosoServer!FEngineLoop::Tick() [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:5915]
0x000000000bc06a3a VirtuosoServer!GuardedMain(char16_t const*) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Launch/Private/Launch.cpp:182]
0x000000000bb6c04d VirtuosoServer!CommonUnixMain(int, char**, int ()(char16_t const), void (*)()) [U:/Source/UE5Git/UnrealEngine/Engine/Source/./Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:269]
0x00007f6b9416f13a libc.so.6!__libc_start_main(+0xe9)
0x0000000005241029 VirtuosoServer!_start()

2

Hi @VirtuosoCEO,

I’m afraid the team hasn’t done much work with multiplayer, so I’m not sure how much we can help. But it would be useful overall to provide more info, such as:

  • Unreal Engine and Cesium for Unreal versions
  • What datasets you’re using in the level
  • A description of when the crash happens. Are there certain conditions or events that are more likely to cause the crash?

Thanks!

3

I am a using a source build of Unreal 5.4.4
Cesium GitHub Release version of 2.13.3
I am using Google Maps 3D Photorealistic Tiles.

It just crashes immediately on the load of the level, I assume when initially trying to load any data.
Linux servers don’t have a renderer and are just command line based, so it may have something to do with that

4

Hi @VirtuosoCEO,

Thanks for getting back to us. From your description, it sounds like you’re trying to render the data on the server itself, rather than client-side – is that correct? Or have I misunderstood?

It might help to talk through your multiplayer logic here. Depending on the scope of your game (e.g., if you’re hosting people around the entire globe), it might be unideal to have a single server instance of the 3D Tileset. But I acknowledge that doing client-side loading would greatly increase the number of root requests you’re making if you use Google Photorealistic 3D Tiles, which wouldn’t be great either…

5

Yes, this is the case.

In theory, the server doesn’t need to render it. I am using it as background geo/detail, however if its present at all, it causes a crash.

In the future, I’d like to use it as a gameplay element as well, so I know I will run into it then. If the geo/collision is recognized by the server regardless, the collision will cause mishaps.

I am using Photorealistic Tiles. I know costs may be intense, but for now, I try to get the basics to work and will worry about cost optimizations later

6

@VirtuosoCEO do you have any other plugins enabled on the server? It’s hard to be sure, but that call stack looks like the sort of thing we see when some other plugin also uses RapidJSON, and its version is not the same as the one used in Cesium for Unreal.

7

If I search my project solution and files for RapidJSON, Cesium is the only result other than within Unreal Source as a ThirdParty library.

If I delete the tiles, it works, so I am unsure if its anything else. Could certainly be related to rendering since the Linux Server is headless/without GPU usage.

I have a fellow dev who has asked me this: “is QueuedScheduler a dispatch? you may have called rapidjson in a dispatch and the rapid json node you are using has a null pointer.”

8

By “some other plugin that uses RapidJSON”, I also mean to include plugins that are shipped with Unreal Engine. In fact, that’s the most common case where this comes up. I think it’s the nDisplay plugin that has caused this sort of conflict in the past.

If I delete the tiles, it works, so I am unsure if its anything else. Could certainly be related to rendering since the Linux Server is headless/without GPU usage.

Yeah, so the crash happens when a tileset is updating. So if you don’t have any tilesets, you will have no crash. From there, it gets confusing. There’s this part of the call stack:

0x00000000107cf996 VirtuosoServer!UnknownFunction()

It’s not clear to me why our main thread task dispatcher would be calling into “your” function. i.e., code that does not appear to be part of Cesium for Unreal. But then that unknown function calls into RapidJSON, and that’s where everything falls apart. It could be falling apart because that code - whatever it is! - is simply using RapidJSON wrong. It could also be using RapidJSON correctly, but is a victim of that “conflicting RapidJSON version” problem I mentioned previously, causing corruption of the stack or mix-up of function parameters.

I think figuring out what that UnknownFunction actually is could be a good first step. Can you run a Debug build on the server and see if you get a better call stack?

“is QueuedScheduler a dispatch? you may have called rapidjson in a dispatch and the rapid json node you are using has a null pointer.

Sorry, I’m not sure what this means.

1 Like
9

I found a temporary solution.

This would not suffice for interactivity, but as far as visual only use, I put all of the Cesium related actors in a sublevel, and on begin play in the Persistent Level Blueprint, check if its a dedicated server, and if not, then load the Cesium sublevel.

I can dig deeper soon. This actually is a development build because AWS for some reason doesn’t allow use of Shipping builds, or at least within GameLift, they all immediately crash no matter what, so our production servers are Development builds. Is Debug even further than development? If so, I can work on that

10

This actually is a development build because AWS for some reason doesn’t allow use of Shipping builds, or at least within GameLift, they all immediately crash no matter what, so our production servers are Development builds.

Hmm that’s interesting / suspicious. Are you saying that shipping builds crash even without any Cesium included at all?

Is Debug even further than development? If so, I can work on that

Yeah, the “DebugGame” configuration should be a bit more debuggable because it disables certain optimizations, like inlining. So you might get a bit more information out of it. It’s worth a try.

11

It could be due to me using the AWS SDK plugin, which I am advised by this from the plugin devs. May not be all builds, but with this plugin it is true.

I will look into trying a DebugGame server this week