Crash on connecting to server

Cesium for Unreal
1

Hi,

We have noticed a consistent crash when trying to connect to the dedicated server while Cesium is in the process of loading tiles. Is there a dedicated workflow to work around this issue?

Here’s the crashlog:

2024-01-16T19:39:45.955914000Z,"0x0000000007f0e8b3 [REDACTED]!FAsyncLoadingThread2::TickAsyncLoadingFromGameThread(FAsyncLoadingThreadState2&, bool, bool, double, int, bool&)()"," [REDACTED]!FAsyncLoadingThread2::TickAsyncLoadingFromGameThread(FAsyncLoadingThreadState2&, bool, bool, double, int, bool&)()"
2024-01-16T19:39:45.955920000Z,0x0000000007f14ac0 [REDACTED]!FAsyncLoadingThread2::FlushLoading(int)()," [REDACTED]!FAsyncLoadingThread2::FlushLoading(int)()"
2024-01-16T19:39:45.955926000Z,0x0000000007f16294 [REDACTED]!FlushAsyncLoading(int)()," [REDACTED]!FlushAsyncLoading(int)()"
2024-01-16T19:39:45.955932000Z,"0x0000000008198d0b [REDACTED]!LoadPackageInternal(UPackage*, FPackagePath const&, unsigned int, FLinkerLoad*, FArchive*, FLinkerInstancingContext const*, FPackagePath const*)()"," [REDACTED]!LoadPackageInternal(UPackage*, FPackagePath const&, unsigned int, FLinkerLoad*, FArchive*, FLinkerInstancingContext const*, FPackagePath const*)()"
2024-01-16T19:39:45.955954000Z,"0x000000000819a1c2 [REDACTED]!LoadPackage(UPackage*, FPackagePath const&, unsigned int, FArchive*, FLinkerInstancingContext const*, FPackagePath const*)()"," [REDACTED]!LoadPackage(UPackage*, FPackagePath const&, unsigned int, FArchive*, FLinkerInstancingContext const*, FPackagePath const*)()"
2024-01-16T19:39:45.955960000Z,"0x0000000008196db6 [REDACTED]!LoadPackage(UPackage*, char16_t const*, unsigned int, FArchive*, FLinkerInstancingContext const*)()"," [REDACTED]!LoadPackage(UPackage*, char16_t const*, unsigned int, FArchive*, FLinkerInstancingContext const*)()"
2024-01-16T19:39:45.955965000Z,"0x0000000008194c7b [REDACTED]!ResolveName(UObject*&, FString&, bool, bool, unsigned int, FLinkerInstancingContext const*)()"," [REDACTED]!ResolveName(UObject*&, FString&, bool, bool, unsigned int, FLinkerInstancingContext const*)()"
2024-01-16T19:39:45.955971000Z,"0x0000000008197304 [REDACTED]!StaticLoadObjectInternal(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"," [REDACTED]!StaticLoadObjectInternal(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"
2024-01-16T19:39:45.955977000Z,"0x0000000008197719 [REDACTED]!StaticLoadObjectInternal(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"," [REDACTED]!StaticLoadObjectInternal(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"
2024-01-16T19:39:45.955983000Z,"0x0000000008179f2e [REDACTED]!StaticLoadObject(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"," [REDACTED]!StaticLoadObject(UClass*, UObject*, char16_t const*, char16_t const*, unsigned int, UPackageMap*, bool, FLinkerInstancingContext const*)()"
2024-01-16T19:39:45.955989000Z,0x0000000004bc60c1 [REDACTED]!FglTFRuntimeParser::LoadAndFillBaseMaterials() [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:360]," [REDACTED]!FglTFRuntimeParser::LoadAndFillBaseMaterials() [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:360]"
2024-01-16T19:39:45.955995000Z,"0x0000000004bc6e28 [REDACTED]!FglTFRuntimeParser::FglTFRuntimeParser(TSharedRef<FJsonObject, (ESPMode)1>, UE::Math::TMatrix<double> const&, float) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:511]"," [REDACTED]!FglTFRuntimeParser::FglTFRuntimeParser(TSharedRef<FJsonObject, (ESPMode)1>, UE::Math::TMatrix<double> const&, float) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:511]"
2024-01-16T19:39:45.956025000Z,"0x0000000004bc5ccd [REDACTED]!FglTFRuntimeParser::FromString(FString const&, FglTFRuntimeConfig const&, TSharedPtr<FglTFRuntimeZipFile, (ESPMode)1>) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:275]"," [REDACTED]!FglTFRuntimeParser::FromString(FString const&, FglTFRuntimeConfig const&, TSharedPtr<FglTFRuntimeZipFile, (ESPMode)1>) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:275]"
2024-01-16T19:39:45.956031000Z,"0x0000000004bc5aaf [REDACTED]!FglTFRuntimeParser::FromBinary(unsigned char const*, long long, FglTFRuntimeConfig const&, TSharedPtr<FglTFRuntimeZipFile, (ESPMode)1>) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:345]"," [REDACTED]!FglTFRuntimeParser::FromBinary(unsigned char const*, long long, FglTFRuntimeConfig const&, TSharedPtr<FglTFRuntimeZipFile, (ESPMode)1>) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:345]"
2024-01-16T19:39:45.956036000Z,"0x0000000004bc3f39 [REDACTED]!FglTFRuntimeParser::FromData(unsigned char const*, long long, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:245]"," [REDACTED]!FglTFRuntimeParser::FromData(unsigned char const*, long long, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeParser.cpp:245]"
2024-01-16T19:39:45.956042000Z,"0x0000000004bfcd43 [REDACTED]!UglTFRuntimeAsset::LoadFromData(unsigned char const*, long long, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeAsset.cpp:97]"," [REDACTED]!UglTFRuntimeAsset::LoadFromData(unsigned char const*, long long, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeAsset.cpp:97]"
2024-01-16T19:39:45.956048000Z,"0x0000000004c49ad2 [REDACTED]!UglTFRuntimeFunctionLibrary::glTFLoadAssetFromData(TArray<unsigned char, TSizedDefaultAllocator<32> > const&, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeFunctionLibrary.cpp:171]"," [REDACTED]!UglTFRuntimeFunctionLibrary::glTFLoadAssetFromData(TArray<unsigned char, TSizedDefaultAllocator<32> > const&, FglTFRuntimeConfig const&) [/tmp/project/Plugins/glTFRuntime/Source/glTFRuntime/Private/glTFRuntimeFunctionLibrary.cpp:171]"
2024-01-16T19:39:45.970736000Z,"0x0000000004c4a8a2 [REDACTED]!decltype(auto) UE::Core::Private::Tuple::TTupleBase<TIntegerSequence<unsigned int, 0u, 1u>, FglTFRuntimeHttpResponse, FglTFRuntimeConfig>::ApplyAfter<UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3&, TSharedPtr<IHttpRequest, (ESPMode)1>&, TSharedPtr<IHttpResponse, (ESPMode)1>&, bool&>(UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3&&&, TSharedPtr<IHttpRequest, (ESPMode)1>&&&, TSharedPtr<IHttpResponse, (ESPMode)1>&&&, bool&&&) const [/home/ue4/UnrealEngine/Engine/Source/Runtime/Core/Public/Templates/Tuple.h:321]"," [REDACTED]!decltype(auto) UE::Core::Private::Tuple::TTupleBase<TIntegerSequence<unsigned int, 0u, 1u>, FglTFRuntimeHttpResponse, FglTFRuntimeConfig>::ApplyAfter<UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3&, TSharedPtr<IHttpRequest, (ESPMode)1>&, TSharedPtr<IHttpResponse, (ESPMode)1>&, bool&>(UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3&&&, TSharedPtr<IHttpRequest, (ESPMode)1>&&&, TSharedPtr<IHttpResponse, (ESPMode)1>&&&, bool&&&) const [/home/ue4/UnrealEngine/Engine/Source/Runtime/Core/Public/Templates/Tuple.h:321]"
2024-01-16T19:39:45.970752000Z,"0x0000000004c4a817 [REDACTED]!TBaseFunctorDelegateInstance<void (TSharedPtr<IHttpRequest, (ESPMode)1>, TSharedPtr<IHttpResponse, (ESPMode)1>, bool), FDefaultDelegateUserPolicy, UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3, FglTFRuntimeHttpResponse, FglTFRuntimeConfig>::ExecuteIfSafe(TSharedPtr<IHttpRequest, (ESPMode)1>, TSharedPtr<IHttpResponse, (ESPMode)1>, bool) const [/home/ue4/UnrealEngine/Engine/Source/Runtime/Core/Public/Delegates/DelegateInstancesImpl.h:744]"," [REDACTED]!TBaseFunctorDelegateInstance<void (TSharedPtr<IHttpRequest, (ESPMode)1>, TSharedPtr<IHttpResponse, (ESPMode)1>, bool), FDefaultDelegateUserPolicy, UglTFRuntimeFunctionLibrary::glTFLoadAssetFromUrl(FString const&, TMap<FString, FString, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FString, FString, false> > const&, FglTFRuntimeHttpResponse, FglTFRuntimeConfig const&)::$_3, FglTFRuntimeHttpResponse, FglTFRuntimeConfig>::ExecuteIfSafe(TSharedPtr<IHttpRequest, (ESPMode)1>, TSharedPtr<IHttpResponse, (ESPMode)1>, bool) const [/home/ue4/UnrealEngine/Engine/Source/Runtime/Core/Public/Delegates/DelegateInstancesImpl.h:744]"
2024-01-16T19:39:45.970758000Z,0x000000000820e9e9 [REDACTED]!FCurlHttpRequest::FinishedRequest()()," [REDACTED]!FCurlHttpRequest::FinishedRequest()()"
2024-01-16T19:39:45.970762000Z,0x00000000082181c1 [REDACTED]!FHttpManager::Tick(float)()," [REDACTED]!FHttpManager::Tick(float)()"
2024-01-16T19:39:45.970769000Z,0x000000000531ec0d [REDACTED]!UnrealAssetAccessor::tick() [/tmp/project/Plugins/CesiumForUnreal/Source/CesiumRuntime/Private/UnrealAssetAccessor.cpp:277]," [REDACTED]!UnrealAssetAccessor::tick() [/tmp/project/Plugins/CesiumForUnreal/Source/CesiumRuntime/Private/UnrealAssetAccessor.cpp:277]"
2024-01-16T19:39:45.970774000Z,0x0000000004cea925 [REDACTED]!ACesium3DTileset::IsReadyForFinishDestroy() [/tmp/project/Plugins/CesiumForUnreal/Source/CesiumRuntime/Private/Cesium3DTileset.cpp:2252]," [REDACTED]!ACesium3DTileset::IsReadyForFinishDestroy() [/tmp/project/Plugins/CesiumForUnreal/Source/CesiumRuntime/Private/Cesium3DTileset.cpp:2252]"
2024-01-16T19:39:45.970780000Z,"0x0000000007fc77c4 [REDACTED]!IncrementalPurgeGarbage(bool, double)()"," [REDACTED]!IncrementalPurgeGarbage(bool, double)()"
2024-01-16T19:39:45.970785000Z,0x0000000007fd6659 [REDACTED]!void UE::GC::CollectGarbageImpl<true>(EObjectFlags)()," [REDACTED]!void UE::GC::CollectGarbageImpl<true>(EObjectFlags)()"
2024-01-16T19:39:45.970790000Z,0x0000000007fd499b [REDACTED]!UE::GC::CollectGarbageFull(EObjectFlags)()," [REDACTED]!UE::GC::CollectGarbageFull(EObjectFlags)()"
2024-01-16T19:39:45.970795000Z,"0x0000000007fc9a41 [REDACTED]!CollectGarbage(EObjectFlags, bool)()"," [REDACTED]!CollectGarbage(EObjectFlags, bool)()"
2024-01-16T19:39:45.970800000Z,0x000000000cf6c94a [REDACTED]!UEngine::TrimMemory()()," [REDACTED]!UEngine::TrimMemory()()"
2024-01-16T19:39:45.970806000Z,"0x000000000cf69fbd [REDACTED]!UEngine::LoadMap(FWorldContext&, FURL, UPendingNetGame*, FString&)()"," [REDACTED]!UEngine::LoadMap(FWorldContext&, FURL, UPendingNetGame*, FString&)()"
2024-01-16T19:39:45.970811000Z,"0x000000000cf688d3 [REDACTED]!UEngine::TickWorldTravel(FWorldContext&, float)()"," [REDACTED]!UEngine::TickWorldTravel(FWorldContext&, float)()"
2024-01-16T19:39:45.970816000Z,"0x000000000c3ff53d [REDACTED]!UGameEngine::Tick(float, bool)()"," [REDACTED]!UGameEngine::Tick(float, bool)()"
2024-01-16T19:39:45.970821000Z,0x0000000004b5a93b [REDACTED]!FEngineLoop::Tick()()," [REDACTED]!FEngineLoop::Tick()()"
2024-01-16T19:39:45.970826000Z,0x0000000004b6206a [REDACTED]!GuardedMain(char16_t const*)()," [REDACTED]!GuardedMain(char16_t const*)()"
2024-01-16T19:39:45.970831000Z,"0x000000000d6ce3b1 [REDACTED]!CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)())()"," [REDACTED]!CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)())()"
2024-01-16T19:39:45.970837000Z,0x00007f39f3c9e083 libc.so.6!__libc_start_main(+0xf2)," libc.so.6!__libc_start_main(+0xf2)"
2024-01-16T19:39:45.970842000Z,0x0000000004b54029 [REDACTED]!_start()," [REDACTED]!_start()"
2024-01-16T19:39:45.970847000Z,,
2024-01-16T19:39:45.970863000Z,Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.," crash handling finished; re-raising signal 11 for the default handler. Good bye."
2

I don’t have a direct answer for you, but I can tell you a little bit about what I see in the call stack and maybe that will help you identify the problem…

It looks like you’re loading a new map, which is triggering an Unreal garbage collection. A Cesium3DTileset is eligible for garbage collection (have you previously removed a tileset?). The way the UE garbage collector works is that it repeatedly polls IsReadyForFinishDestroy on a GC-eligible object until that function returns true, and then it finally deletes it. Cesium3DTileset is implemented to only return true after all of its pending network requests have wrapped up. But it also calls Tick on the FHttpManager in order to move those requests along, so that they can actually finish eventually.

Now, during a tick of the HTTP manager, some code that is totally unrelated to Cesium gets invoked, specifically UglTFRuntimeFunctionLibrary::glTFLoadAssetFromData. Presumably it is also doing network requests, and one is finishing during one of these Cesium-invoke HTTP ticks, and its callback is being invoked.

Where it goes wrong from there, I’m not really sure. What sort of crash is this? An assertion failure? An access violation? Something else? What I can see is that FglTFRuntimeParser is seemingly trying to load a package (asset) and that is somehow failing or crashing. But as I don’t know anything about that library, I can’t guess what might be going wrong.

I don’t see any reason to suspect from this callstack that Cesium is doing anything wrong, though. It’s perhaps a little unusual to tick HTTP requests during a map load, but if that is causing another library to crash then I’d argue the bug is in that other library. So it could be worth raising the question with the developer of UglTFRuntimeFunctionLibrary.

3

I missed the actual assertion line, here it is:

Error: appError called: Assertion failed: !IsGarbageCollecting() [File:./Runtime/CoreUObject/Private/Serialization/AsyncLoading2.cpp] [Line: 6977] 
Assertion failed: !IsGarbageCollecting() [File:./Runtime/CoreUObject/Private/Serialization/AsyncLoading2.cpp] [Line: 6977]

I believe at this point I know what’s happening here, and it does appear to be an issue with the glTFRuntime plugin. These things happen in sequence during the single world tick:
1. Cesium has finished its tile loading and has triggered garbage collection
2. A Http request by the glTFRuntime plugin has finished
3. glTFRuntime tries to load base materials FglTFRuntimeParser::LoadAndFillBaseMaterials
4. Engine function LoadObject asserts, because GC pass is in progress

4

I don’t see any evidence in the call stack that Cesium triggered the garbage collection. It was triggered by a map load, which, in turn, may have been triggered by entering a Cesium sub-level (or it could be a map load unrelated to Cesium). Other than that, your explanation sounds right to me.

5

Cesium must play a role here, because as soon as I remove the Cesium3DTileset actor on BeginPlay on the client while it’s waiting to connect to the server, the crash goes away. So the wording in my previous message was not accurate enough - Cesium has not triggered GC, but might have blocked it from triggering while it was loading tiles. Somehow this creates circumstances for the race condition that results in the glTFRuntime plugin crashing.
I already received a patch from the glTFRuntime creator and am in the process of testing it. So hopefully this will no longer happen.