How to rotate (and scale) a Gaussian Splat in CesiumJS

CesiumJS
1

I have an spz rendered in CesiumJS using the splat-spz-concept branch of the github. I’m able to reposition it fine, but trying to rotate it using the tileset’s modelMatrix appears to be rotating each individual splat instead of the whole ‘model’ - see these pictures

Rotation Screenshot 1
Rotation Screenshot 11918×907 250 KB

Rotation Screenshot 2
Rotation Screenshot 21917×907 250 KB

I need to rotate the ‘parent transform’ per se so that it is no longer perpendicular to the ground. So far I’ve tried modifying root.transform and GaussianSplat3DTileContent’s worldTransform as well as changing the up axis/forward axis with no success. I used this tool to convert the spz into glb: GitHub - javagl/JSpz: Java loader for SPZ

Is there a way to rotate the entire GS? It doesn’t necessarily have to be at runtime, I just need it to load in with the correct transform. (I haven’t gotten scaling to work either but rotation is my more immediate concern)

2

The branch of CesiumJS is still experimental. There are a few things that may still change. And there are a few things that are not yet fully working as expected.

One of these things seems to be: Dynamic adjustments of the coordinate transforms.

When I did a few experiments with a basic splats tileset, I also stumbled over some of these issues. Now, I tried it out based on the current state, and noticed that it does take into account the tileset.modelMatrix, but only under certain conditions. One of the conditions seems to be (roughly) that the modelMatrix has to be set before the splat content is loaded.

At least, I created the following test, which just loads a tileset, assigns its modelMatrix. And this does seem to take into account the rotationMatrix:

const viewer = new Cesium.Viewer("cesiumContainer");

// Create the tileset in the viewer
const tileset = viewer.scene.primitives.add(
  await Cesium.Cesium3DTileset.fromUrl(
    "http://localhost:8003/tileset.json", {
    debugShowBoundingVolume: true,
  })
);

// Prepare the model matrix
const modelMatrix = Cesium.Matrix4.clone(Cesium.Matrix4.IDENTITY);

// Transform the tileset to some position on the globe
const transform = Cesium.Transforms.eastNorthUpToFixedFrame(
  Cesium.Cartesian3.fromDegrees(-75.152408, 39.946975, 20)
);
Cesium.Matrix4.multiply(modelMatrix, transform, modelMatrix);

// Apply some rotation
const rotationMatrixX = Cesium.Matrix3.fromRotationX(
  Cesium.Math.toRadians(90), new Cesium.Matrix4());
const rotationMatrix = Cesium.Matrix4.fromRotation(
  rotationMatrixX, new Cesium.Matrix4());
Cesium.Matrix4.multiply(modelMatrix, rotationMatrix, modelMatrix);

// Set the final model matrix
tileset.modelMatrix = modelMatrix;

// Zoom to the tileset, with a small offset so that
// it is fully visible
const offset = new Cesium.HeadingPitchRange(
  Cesium.Math.toRadians(-22.5),
  Cesium.Math.toRadians(-22.5),
  20.0
);
viewer.zoomTo(tileset, offset);

In contrast to that, here is a sandcastle with basic editing components for the model matrix, and one can see that this does not work: It does not seem to take into account changes of the matrix, and if it does (e.g. by modifying the camera/view), then the changes don’t seem to be taken into account correctly.

Sandcastle URL for localhost on SPZ branch

A bit more generally:

There are several other “moving targets” right now, because Gaussian Splats are currently a tremendously active research topic.Specifically: The fundamental question of “How are Gaussian Splats oriented (in terms of axis conventions)?” seems to largely be unanswered right now. There is a related question in an issue about splat file formats that suggests that many tools are using different axis conventions.

For Gaussian Splats in glTF with SPZ the situation should be more clear:

glTF uses the y-up-convention. Whatever the source format of the splats is: In glTF, it has to be y-up. And the KHR_spz_gaussian_splats_compression extension proposal contains a conformance section that says:

When compressing or decompressing the SPZ data to be stored within the glTF, you must specify a Left-Up-Front (LUF ) coordinate system in the SPZ PackOptions or UnpackOptions within the SPZ library. This ensures that the data is compressed and decompressed appropriately for glTF.

This refers to the Coordinate System convention of SPZ. But there might still be data out there that was generated with different conventions.

1 Like
3

Currently, the best way to do this is when you create your glTF file, apply any transforms in the node transformation matrix.

Right now we do a one time transform after the content is loaded, so it won’t update correctly with any model matrix changes. Something of a side effect from taking N tiles and building a single draw call.

This is definitely something we want to support in the future though.

Thanks for checking out the branch. It is experimental, but we do appreciate any feedback.

1 Like
4

In case that you are using the SpzToTileset example, you might be able to perform the required adjustment by setting a different matrix in the glTF root node. For example, when you change this matrix to

{ 0.1f, 0.0f, 0.0f, 0.0f,
  0.0f, 0.1f, 0.0f, 0.0f, 
  0.0f, 0.0f, 0.1f, 0.0f, 
  0.0f, 0.0f, 0.0f, 1.0f }

then this should apply a uniform scale factor of 0.1 to the splat. And it should be possible to apply rotations similarly.

(All this is not tested extensively. And as mentioned above: In theory, such coordinate conversions should not be necessary when everybody obeys the coordinate system conventions. The issue Clarify the Coordinate System Definition · Issue #14 · nianticlabs/spz · GitHub has been addressed with the small section in the README.md, but only recently - so maybe there is still data out there that uses different conventions)

5

Apologies, I misspoke yesterday. The best place is actually the transform matrix in the tileset, not the glTF.

Quick note on scaling specifically. We actually strip the scaling factor from the transform matrix in the shader. This caused side effects of affecting the scale of each splat when projecting into 2D space rather than the overall scale of the tileset. There is a separate scaling factor that is planned to be reactivated as part of a tileset style.

6

Well, I should have tested it :laughing:

In doubt, the transform can also easily be added on the tileset JSON level. That “Spz-To-Tileset” example uses some wierd, inlined, quick-and-dirty hard-wired tileset JSON, so such a transform could be added as a transform: [ ... ] property in that tileset JSON (even though, at this point, manually editing the JSON could be easier, unless the same transform has to be applied for many SPZ input files…)

7

I just added a writing functionality to JSpz, and there now is a JSpzTransform example that shows how to do a coordinate conversion. (You’d still have to find out which conversion you need, exactly, but maybe that’s useful nevertheless)

1 Like
8

Thank you for the replies. I’ve been able to set rotation and scale with the root transform matrix after fixing the file’s coordinate system