Approx 200k Entities - Out Of Memory Error - KMZ Source

CesiumJS
1

I am currently running CesiumJS 1.99 and am attempting to load in a large KMZ (20 MB) into Cesium. The KMZ consists of polylines, billboards, labels, and polygons. Loading in 75% of the KMZ I end up with 195,292 entities.

When loading the entire KMZ all at once I receive a out of memory error in Chrome, Firefox, and Edge.
I then tried to split it up into 2 KMZs and load them one after another. The first one loads fine but when loading in the second I receive the out of memory error again.

I then tried to split it into 20 chunks and load them one at a time. This works if I wait very long between loading each KMZ (about 2 minutes). I suffer no FPS degradation and it is quite smooth. If I try to load them one after another I run into the memory error at about the same spot.

I am running this self hosted. I did check CesiumION to see if it would help, however it said that my KMZ is not supported for tiling:

We were unable to process your data due to the following issue:

Error: Input does not contain any eligible features for tiling.

Read about what data formats are supported on Cesium ion.

The error seems to happen when the main thread hits around ~3800 MB.If I move slowly it seems like the GC can catch up and remove some unused things.

I have tried a number of ways to optimize the import but as of now I am doing:

First I set up my Cesium viewer. I have tried to optimize my settings here to save memory, such as requestRenderMode, tried requesting WebGL 2.0, etc. The options below are just where I ended up after a lot of trial and error:

  const viewer = await new Cesium.Viewer(`threeDViewer`, {
    imageryProvider: new Cesium.OpenStreetMapImageryProvider({
      url: 'https://a.tile.openstreetmap.org/',
    }),
    vrButton: true,
    timeline: false,
    animation: false,
    shouldAnimate: false,
    credits: false,
    scene3DOnly: true,
    fullscreenElement: "threeDViewer",
    allowDataSourcesToSuspendAnimation: true,
    requestRenderMode : true,
    maximumRenderTimeChange : Infinity,
    contextOptions: {
      requestWebgl1: true,
    },
  })

Then I load in my KMZ like so.

viewer.entities.suspendEvents()
  let rasterSource = await new Cesium.KmlDataSource.load(
    `${process.env.NEW_URL}/api/staticFiles/${dataset}/kml/${rasterFile}`
  )
for (let i = 0; i < rasterSource.entities.values.length; i++) {
    if (Cesium.defined(rasterSource.entities.values[i].rectangle)) {
      rasterSource.entities.values[i].rectangle.material.color = rasterColor
    }
    if (Cesium.defined(rasterSource.entities.values[i].polyline)) {
      rasterSource.entities.values[i].polyline.heightReference = Cesium.HeightReference.RELATIVE_TO_GROUND;
    }
      if (Cesium.defined(rasterSource.entities.values[i].label) === true && Cesium.defined(rasterSource.entities.values[i].billboard) === true && dataset !== 'NEET') {
      console.log(rasterSource.entities.values[i])
      rasterSource.entities.values[i].billboard.verticalOrigin = Cesium.VerticalOrigin.BOTTOM
      rasterSource.entities.values[i].label.verticalOrigin = Cesium.VerticalOrigin.BOTTOM
      let objectsToExclude = [rasterSource.entities.values[i]];
      let position = rasterSource.entities.values[i].position.getValue(viewer.clock.currentTime);
      rasterSource.entities.values[i].position = viewer.scene.clampToHeight(position, objectsToExclude);
      //rasterSource.entities.values[i].label.heightReference = Cesium.HeightReference.RELATIVE_TO_GROUND;
    }
    await viewer.entities.add(rasterSource.entities.values[i])
}
await rasterSource.destroy()
  viewer.scene.requestRender();
  viewer.entities.resumeEvents()

Destroying the datasource at the end after adding it to the entities seems to let the program get a little further.

I have tried other ways of adding the data, such as just doing the following by adding the datasource.

viewer.dataSources.add(rasterSource)

But this had worse performance.

I also attempted to add the objects through the primitive API instead of the entity API. I had a few issues
with this:

The FPS took a major hit. Even when loading 1/20 of the objects with the entity API the rendering did not seem to be nearly as smooth.

I was able to load in the entire object piece by piece but it required quite a bit of manipulation, such as:

  1. Declare a primitive collection.
  2. Declare a billboard collection.
  3. Declare a polyline collection.
  4. Declare an array for instances of geometry.
  5. Loop through the datasource, finding the entity type, recreating it with the .add() functionality of its given type, attempting to read the properties of the datasource version and apply to the primitive version.
  6. Once complete add the collections with the viewer.scene.primitives.add() methodology.

I had trouble taking some of the datasource entity properties and applying them to the primitive version. The big one was coloring on the polylines/polyline graphics (I am unsure specifically what the difference is between the two?) and labels not coming in with billboards.

I have attempted to use both Firefox and Chrome’s memory profilers but they just end up crashing.

Here is a sample of one of the polyline/line string entities in the KMZ. You can see the lines are quite complex:

image
image899×449 131 KB

I’m unsure if there are other optimizations available, maybe something I am over looking, or further recommendations.

Here are the JSVM memory usages in the Chrome devtools after loading 15/20 of the split up KMZs:

image
image653×1197 64.3 KB

There is also a large 3D Tileset primitive in this but I have not had any issues with it. I am (attempting to) drawing the KMZ entities on top of the 3D Tileset.