Crash in rapidjson on Android with latest 1.12.2 plugin

[carlrealvr]

We just tried updating the Cesium plugin to the latest version 1.12.2 in unreal marketplace and getting a crash in rapidjson. We’re on Unreal Engine 4.27.2 and Android. Here’s the stack trace

05-04 14:15:17.872 26386 26428 F libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10000 in tid 26428 (TaskGraphNP 0), pid 26386 (main)
05-04 14:15:18.056 26764 26764 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-04 14:15:18.056 26764 26764 F DEBUG   : Revision: '0'
05-04 14:15:18.056 26764 26764 F DEBUG   : ABI: 'arm64'
05-04 14:15:18.056 26764 26764 F DEBUG   : Timestamp: 2022-05-04 14:15:18-0700
05-04 14:15:18.056 26764 26764 F DEBUG   : pid: 26386, tid: 26428, name: TaskGraphNP 0  >>> app.test.local <<<
05-04 14:15:18.056 26764 26764 F DEBUG   : uid: 10272
05-04 14:15:18.056 26764 26764 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10000
05-04 14:15:18.057 26764 26764 F DEBUG   :     x0  00000078aa635cc0  x1  000000790327c5e8  x2  0000007807469f80  x3  0000000000000001
05-04 14:15:18.058 26764 26764 F DEBUG   :     x4  00000078aa7db003  x5  00000078aa635cab  x6  0000000000000075  x7  000000000000006c
05-04 14:15:18.058 26764 26764 F DEBUG   :     x8  0000000000000024  x9  00000078aa636000  x10 00000078aa7db01d  x11 0000000000000022
05-04 14:15:18.058 26764 26764 F DEBUG   :     x12 0000000100002600  x13 0000000000000022  x14 fffffffffc000000  x15 000000000000d819
05-04 14:15:18.058 26764 26764 F DEBUG   :     x16 00000078a77319e0  x17 00000079994010c0  x18 000000788da3c000  x19 000000790327c5e8
05-04 14:15:18.058 26764 26764 F DEBUG   :     x20 00000078aa635cc0  x21 0000007807469f80  x22 0000000000000020  x23 0000000000010000
05-04 14:15:18.058 26764 26764 F DEBUG   :     x24 000000790327d020  x25 00000078aa7db01b  x26 00000000ffff2400  x27 000000790327c778
05-04 14:15:18.058 26764 26764 F DEBUG   :     x28 0000000000000074  x29 000000790327c5d0
05-04 14:15:18.058 26764 26764 F DEBUG   :     sp  000000790327c5a0  lr  000000789f6f52fc  pc  000000789f6f3868
05-04 14:15:18.215 26764 26764 F DEBUG   : 
05-04 14:15:18.215 26764 26764 F DEBUG   : backtrace:
05-04 14:15:18.215 26764 26764 F DEBUG   :       #00 pc 000000000b1c6868  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>>::SetStringRaw(rapidjson::GenericStringRef<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&)+124) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #01 pc 000000000b1c82f8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #02 pc 00000000118e8824  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #03 pc 00000000118e8ad8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #04 pc 00000000118e8ad8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #05 pc 00000000118e7f94  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #06 pc 00000000118e7e3c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #07 pc 00000000118f9280  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #08 pc 000000001199ca48  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #09 pc 000000000be0f700  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (_ZN10TGraphTaskI15FAsyncGraphTaskE11ExecuteTaskER6TArrayIP14FBaseGraphTask22TSizedDefaultAllocatorILi32EEEN13ENamedThreads4TypeE+796) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #10 pc 000000000be08c24  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::ProcessTasks()+848) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #11 pc 000000000be07f4c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::ProcessTasksUntilQuit(int)+88) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #12 pc 000000000be07d3c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::Run()+60) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #13 pc 000000000bf27914  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FRunnableThreadPThread::Run()+164) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #14 pc 000000000be0546c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FRunnableThreadPThread::_ThreadProc(void*)+84) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #15 pc 00000000000d5884  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: a6e0ca3a9989715134d1d1e2126e0f7a)
05-04 14:15:19.754  1046 26771 I chatty  : uid=1000 system_server expire 1 line

it’s happening for any data set we tried, here’s the last few lines of our app log before it crashed for Denver 354307

[2022.05.04-22.45.06:526][940]LogHttp: 0x7d3aa41750: Starting GET request to URL='https://api.cesium.com/v1/assets/2/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:881][ 75]LogCesium: Warning: SetGeoreferenceOrigin was renamed to SetGeoreferenceOriginLongitudeLatitudeHeight.
[2022.05.04-22.45.09:881][ 75]PIE: Warning: Mobility of /Game/Test/Maps/Level_Cesium.Level_Cesium:PersistentLevel.CesiumBaseCity_4 : Tileset has to be 'Movable' if you'd like to move. 
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7cbb6d09a0: Starting GET request to URL='https://api.cesium.com/v1/assets/1/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7d0af664e0: Starting GET request to URL='https://api.cesium.com/v1/assets/2/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7cb156f250: Starting GET request to URL='https://api.cesium.com/v1/assets/354307/endpoint?access_token=eyJhbG...'

It’s happening for other 3D tilesets as well, Melbourne 69380, Washington State 57590, and others that we are loading externally not hosted on Cesium Ion

[Kevin_Ring]

Hi @carlrealvr,

When we’ve seen RapidJSON-related crashes in the past, it has been caused by the application or some other plugin also using RapidJSON, and a different version of it from the one that Cesium for Unreal uses. When these two different versions get linked into a single executable, the inconsistency can lead to crashes.

Can you confirm if your application or one of its plugins is using RapidJSON? If so, are you able to use Cesium for Unreal’s version? That’s likely the easiest fix.

Unfortunately, this is an extremely challenging problem to solve in the general case. I have two ideas for doing this, but neither one is great:

1. Rename/namespace all of RapidJSON so that Cesium for Unreal gets its own private copy.
2. Configure Cesium for Unreal to always be built into a separate .dll / .so, even in shipping builds. If this is even possible.

Kevin

[Kevin_Ring]

Here’s another thread talking about RapidJSON-related crashes:

Built Project Crash at rapidjson Call v1.8.1

Hi everyone, we recently just compiled CesiumForUnreal v1.8.1 to work with. However, it is resulting in a crash ONLY in built projects. The error only occurs when a World Terrain Tileset is present in the scene and the log indicates it is crashing during a rapidjson call. We test multiple projects all were correctly work/still correctly work with the previous v1.7.0 release. The project runs correctly in PIE and in Standalone but crash at startup of the built package. Has anyone had this ha…

[carlrealvr]

We’ve identified it’s conflicting with one of our plugins on an older RapidJSON version. We’re thinking of taking step 1 or 2 in the end of your suggestion above but we’re not sure how to do it. Can you guys update your RapidJSON in a way that doesn’t conflict with other versions? It seems like that’s creating a lot of issues based on our experience and that of others.

[Kevin_Ring]

I was suggesting that (1) and (2) are things we at Cesium would have to do, I wasn’t suggested that you could do them. However, modifying a third-party library so that another version can exist side-by-side is very challenging. I don’t think we’ll be able to do it soon.

Which is why I suggested the best path is to get the versions in sync. The RapidJSON used by Cesium for Unreal can be found in the Source\ThirdParty\include\rapidjson directory of the plugin. You can find the exact version we’re using by looking at the cesium-native submodule: cesium-native/extern at main · CesiumGS/cesium-native · GitHub

As of this writing, it is commit fd3dc29a5c2852df569e1ea81dbde2c412ac5051 (RapidJSON hasn’t done an official release since 2016).

Kevin

[carlrealvr]

Hi @Kevin_Ring
So in the previous version 1.6.0 (the last one before undergoing the current upgrade) we had exposed a parameter called Exponent, using it to control one of the levers (can’t remember if that’s tiles or something else but it was important). Once again, since we’re in mobile android we have very special needs, and the system automated features cannot be used, we had to customize it to our special use case. This is based on one of your previous answers for version 1.6.0. Now we’ve upgraded to version 1.14.0 and whenever we add this exponent under TilesetOptions, we are able to compile but not to use the Cesium3DTileset object in our levels. Question: How can we reintroduce this exponent?

[Kevin_Ring]

Hmm I’m not quite sure. Can you share the exact code that used to work but doesn’t anymore? And what error you’re getting? Or perhaps to my previous post where I suggested this approach?

[carlrealvr]

@Kevin_Ring

I believe the original post was here:

Better control over how many tiles are loaded at any given time

A few reasons for why it is difficult (for me) to give focussed, goal-oriented advice: It is not yet entirely clear what the actual, underlying reason for the error is. You suggested that it is related to the system running out of memory, and that may indeed be the case. But we don’t have the possibility to analyze this in the actual environment where the problem appears. Therefore, every attempt to solve the problem will involve some guesswork. And we have already seen examples of such guesswo…

and we also used this as a reference Allow non-linear screen space error · Issue #342 · CesiumGS/cesium-native · GitHub

[Kevin_Ring]

Ok thanks @carlrealvr, but I’m still not sure how to help. I understand you’ve added some code to somehow adjust tile loading behavior, but I don’t know exactly what that code is. And now in the latest versions of Cesium for Unreal, that code somehow causes the tileset to fail, but you haven’t said how it fails. So I’m going to need a lot more information in order to help you solve this.

If the original problem was high memory usage leading to crashes, have you tried the latest version of Cesium for Unreal without any modifications? The plugin’s memory behavior has improved substantially since v1.6.0, so if we’re lucky perhaps the modifications are no longer needed.

Kevin

[carlrealvr]

Thanks @Kevin_Ring seems to work better indeed once we updated, and for now also stable without the custom change to Exponent.

[carlrealvr]

@Kevin_Ring Getting a pesky crash though, wondering if you can take a look here (created as a new topic) Crashing when a Cesium title loads and we switch levels