Crash in rapidjson on Android with latest 1.12.2 plugin

We just tried updating the Cesium plugin to the latest version 1.12.2 in unreal marketplace and getting a crash in rapidjson. We’re on Unreal Engine 4.27.2 and Android. Here’s the stack trace

05-04 14:15:17.872 26386 26428 F libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10000 in tid 26428 (TaskGraphNP 0), pid 26386 (main)
05-04 14:15:18.056 26764 26764 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-04 14:15:18.056 26764 26764 F DEBUG   : Revision: '0'
05-04 14:15:18.056 26764 26764 F DEBUG   : ABI: 'arm64'
05-04 14:15:18.056 26764 26764 F DEBUG   : Timestamp: 2022-05-04 14:15:18-0700
05-04 14:15:18.056 26764 26764 F DEBUG   : pid: 26386, tid: 26428, name: TaskGraphNP 0  >>> app.test.local <<<
05-04 14:15:18.056 26764 26764 F DEBUG   : uid: 10272
05-04 14:15:18.056 26764 26764 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10000
05-04 14:15:18.057 26764 26764 F DEBUG   :     x0  00000078aa635cc0  x1  000000790327c5e8  x2  0000007807469f80  x3  0000000000000001
05-04 14:15:18.058 26764 26764 F DEBUG   :     x4  00000078aa7db003  x5  00000078aa635cab  x6  0000000000000075  x7  000000000000006c
05-04 14:15:18.058 26764 26764 F DEBUG   :     x8  0000000000000024  x9  00000078aa636000  x10 00000078aa7db01d  x11 0000000000000022
05-04 14:15:18.058 26764 26764 F DEBUG   :     x12 0000000100002600  x13 0000000000000022  x14 fffffffffc000000  x15 000000000000d819
05-04 14:15:18.058 26764 26764 F DEBUG   :     x16 00000078a77319e0  x17 00000079994010c0  x18 000000788da3c000  x19 000000790327c5e8
05-04 14:15:18.058 26764 26764 F DEBUG   :     x20 00000078aa635cc0  x21 0000007807469f80  x22 0000000000000020  x23 0000000000010000
05-04 14:15:18.058 26764 26764 F DEBUG   :     x24 000000790327d020  x25 00000078aa7db01b  x26 00000000ffff2400  x27 000000790327c778
05-04 14:15:18.058 26764 26764 F DEBUG   :     x28 0000000000000074  x29 000000790327c5d0
05-04 14:15:18.058 26764 26764 F DEBUG   :     sp  000000790327c5a0  lr  000000789f6f52fc  pc  000000789f6f3868
05-04 14:15:18.215 26764 26764 F DEBUG   : 
05-04 14:15:18.215 26764 26764 F DEBUG   : backtrace:
05-04 14:15:18.215 26764 26764 F DEBUG   :       #00 pc 000000000b1c6868  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>>::SetStringRaw(rapidjson::GenericStringRef<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&)+124) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #01 pc 000000000b1c82f8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #02 pc 00000000118e8824  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #03 pc 00000000118e8ad8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #04 pc 00000000118e8ad8  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #05 pc 00000000118e7f94  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #06 pc 00000000118e7e3c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #07 pc 00000000118f9280  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #08 pc 000000001199ca48  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #09 pc 000000000be0f700  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (_ZN10TGraphTaskI15FAsyncGraphTaskE11ExecuteTaskER6TArrayIP14FBaseGraphTask22TSizedDefaultAllocatorILi32EEEN13ENamedThreads4TypeE+796) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #10 pc 000000000be08c24  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::ProcessTasks()+848) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #11 pc 000000000be07f4c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::ProcessTasksUntilQuit(int)+88) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #12 pc 000000000be07d3c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FTaskThreadAnyThread::Run()+60) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #13 pc 000000000bf27914  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FRunnableThreadPThread::Run()+164) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #14 pc 000000000be0546c  /data/app/app.test.local-ZctD6U57puswVm1LoF_5HA==/lib/arm64/libUE4.so (FRunnableThreadPThread::_ThreadProc(void*)+84) (BuildId: 1fdbccb6909a14a1983c9869a182f2a5c0a7d08e)
05-04 14:15:18.215 26764 26764 F DEBUG   :       #15 pc 00000000000d5884  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: a6e0ca3a9989715134d1d1e2126e0f7a)
05-04 14:15:19.754  1046 26771 I chatty  : uid=1000 system_server expire 1 line

it’s happening for any data set we tried, here’s the last few lines of our app log before it crashed for Denver 354307

[2022.05.04-22.45.06:526][940]LogHttp: 0x7d3aa41750: Starting GET request to URL='https://api.cesium.com/v1/assets/2/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:881][ 75]LogCesium: Warning: SetGeoreferenceOrigin was renamed to SetGeoreferenceOriginLongitudeLatitudeHeight.
[2022.05.04-22.45.09:881][ 75]PIE: Warning: Mobility of /Game/Test/Maps/Level_Cesium.Level_Cesium:PersistentLevel.CesiumBaseCity_4 : Tileset has to be 'Movable' if you'd like to move. 
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7cbb6d09a0: Starting GET request to URL='https://api.cesium.com/v1/assets/1/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7d0af664e0: Starting GET request to URL='https://api.cesium.com/v1/assets/2/endpoint?access_token=eyJhbG...'
[2022.05.04-22.45.09:884][ 76]LogHttp: 0x7cb156f250: Starting GET request to URL='https://api.cesium.com/v1/assets/354307/endpoint?access_token=eyJhbG...'

It’s happening for other 3D tilesets as well, Melbourne 69380, Washington State 57590, and others that we are loading externally not hosted on Cesium Ion

Hi @carlrealvr,

When we’ve seen RapidJSON-related crashes in the past, it has been caused by the application or some other plugin also using RapidJSON, and a different version of it from the one that Cesium for Unreal uses. When these two different versions get linked into a single executable, the inconsistency can lead to crashes.

Can you confirm if your application or one of its plugins is using RapidJSON? If so, are you able to use Cesium for Unreal’s version? That’s likely the easiest fix.

Unfortunately, this is an extremely challenging problem to solve in the general case. I have two ideas for doing this, but neither one is great:

  1. Rename/namespace all of RapidJSON so that Cesium for Unreal gets its own private copy.
  2. Configure Cesium for Unreal to always be built into a separate .dll / .so, even in shipping builds. If this is even possible.

Kevin

Here’s another thread talking about RapidJSON-related crashes:

We’ve identified it’s conflicting with one of our plugins on an older RapidJSON version. We’re thinking of taking step 1 or 2 in the end of your suggestion above but we’re not sure how to do it. Can you guys update your RapidJSON in a way that doesn’t conflict with other versions? It seems like that’s creating a lot of issues based on our experience and that of others.

I was suggesting that (1) and (2) are things we at Cesium would have to do, I wasn’t suggested that you could do them. However, modifying a third-party library so that another version can exist side-by-side is very challenging. I don’t think we’ll be able to do it soon.

Which is why I suggested the best path is to get the versions in sync. The RapidJSON used by Cesium for Unreal can be found in the Source\ThirdParty\include\rapidjson directory of the plugin. You can find the exact version we’re using by looking at the cesium-native submodule: cesium-native/extern at main · CesiumGS/cesium-native · GitHub

As of this writing, it is commit fd3dc29a5c2852df569e1ea81dbde2c412ac5051 (RapidJSON hasn’t done an official release since 2016).

Kevin