On-premise tiler errors

Cesium ion
1

Two separate models which I’m trying to convert are failing, each with a different error message.

Model 1; 20gb photogrammetry OBJ. After 30+ hours of tiling, received the following error during Generate Tiles stage:

Using 29 processes, estimate max memory use: 59392 MB
working on 89 models
temp directory is C:\path\to\temp
Check input files [====================] 89/89(100%) 0.0 seconds
Convert models [====================] 89/89(100%) 57.4 seconds
done loading and converting files: 1:01.168 (m:ss.mmm)
target voxel count: 46150
Map density [====================] 178/178(100%) 10.1 seconds
generating density grid: 10.099s
subdividing density grid: 121.058ms
voxel subdivision results: 2527
intermediate bounding boxes: 1019
subtree roots: 1508
Generate Tiles [=                   ] 114/2527(4%) 118068.8 seconds

forked worker terminated unexpectedly: 3221225477

Model 2; 12gb surface mesh (converted from point cloud), tiling as photogrammetry OBJ. Fails during convertion stage:

Using 29 processes, estimate max memory use: 59392 MB
working on 1 models
temp directory is C:\path\to\temp
Check input files [====================] 1/1(100%) 0.0 seconds
Convert models [                    ] 0/1(0%) 0.0 secondsnode:internal/validators:92
      throw new ERR_OUT_OF_RANGE(name, `>= ${min} && <= ${max}`, value);
      ^

RangeError [ERR_OUT_OF_RANGE]: The value of "length" is out of range. It must be >= 0 && <= 2147483647. Received 3323992144
    at Object.write (node:fs:817:5)
    at Object.write (pkg/prelude/bootstrap.js:880:29)
    at writeAll (node:fs:2064:6)
    at node:fs:2126:7
    at FSReqCallback.oncomplete (node:fs:188:23) {
  code: 'ERR_OUT_OF_RANGE'
}


Workerpool Worker terminated Unexpectedly
    exitCode: `1`
    signalCode: `null`
    workerpool.script: `C:\snapshot\swayze\lib\SteelDawnWorker.js`
    spawnArgs: `C:\...\bin\model-tiler.exe,C:\snapshot\swayze\lib\SteelDawnWorker.js`
    spawnfile: `C:\...\bin\model-tiler.exe`
    stdout: `null`
    stderr: `null`

For the second model, I will make several smaller objs from the single 12gb one to see if that makes a difference, as it seems like a file size issue. Still unsure though - I’ll report back on this one.

The first model is made up from 89 smaller objs, we’ve tiled models of this size before FYI. 3221225477 seems to be an NTSTATUS code STATUS_ACCESS_VIOLATION.

Any help will be greatly appreciated.

System details
Processor Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 2.30 GHz (2 processors)
Installed RAM 64.0 GB
System type 64-bit operating system, x64-based processor
Edition Windows 10 Enterprise
Version 21H2
OS build 19044.1889
Experience Windows Feature Experience Pack 120.2212.4180.0
Cores 20
Logical processors 40

100GB of free disk space. Tiling pipeline version 4.5.5

Thanks,
Tom

2

Hi Tom

Thanks for reaching out.

For Model 2 - that is a problem we’ve seen a lot more of recently, and it has to do with the buffer limits of NodeJS. I’m not sure I have great options to share that might readily solve the problem (for either model). I can certainly understand the frustration of waiting that long then then seeing the error.

Would you be able to share these models with us? We can also take a look to see if we are able to debug it in the current pipeline and provide any short-term recommendations.

As we write the new photogrammetry pipeline, we are curating datasets that we test extensively with to ensure that power users like yourself have a fantastic experience using the tilers, and that its stable and performant. Access to these models, if possible, will allow the engineers to constantly test against it and make sure we’re not regressing.

Shehzan

3

Thanks for the quick response @Shehzan_Mohammed,

The first model (error code 3221225477) we are tiling again and haven’t come across the same error as of yet. Now it’s using 9 processes as opposed to 29, estimated max memory use is now 18432MB as opposed to 59392MB. Currently at 6% after 25 hours, so we’ll see how it goes.

Yes we would be happy to send you the raw data - I will send a download link to support@cesium.com and cc yourself. If you have another preferred method for sending this data over, just let me know.

Thanks,
Tom

4

@Shehzan_Mohammed, FYI both have tiled successfully.

The first model tiled in around 1 week~

The second model, we split the raw obj into several smaller objs and it tiled in about 1 hour. Does look like the issue was a memory issue with one large obj.

Thanks,
Tom

5

Hi Tom

Thanks for the update and glad to know it worked out.

If you are still able to send the original data so that we can test it as we work on the new pipeline, that would be very much of interest to us.

Shehzan

6

Hi Shehzan,

I am also experiencing errors with model-tiler 4.5.5 that ran successfully with 4.1.1.

I have sent you an email.

Thanks,
Sam