Sync model from CesiumJS

gtk2k · June 2, 2023, 3:01am

I would like to synchronize the 3D Model (Model Entity) placed with CesiumJS to Cesium for Unity.
How can I reflect the modelMatrix (position/rotation/scale) of the CesiumJS 3D Model to the 3D Model placed in Cesium for Unity?

Kevin_Ring · June 2, 2023, 4:16am

I think I’d start by attaching a CesiumGlobeAnchor component to the model and then set the localToGlobeFixedMatrix property on the component to the same matrix you used in CesiumJS. That may just work. I’m assuming here that your model is not a Cesium3DTileset. Let me know if it is, because the answer will be different.

gtk2k · June 2, 2023, 5:13am

Thank you reply.

(I forgot to tell you one important thing.
I am currently developing a project using Google Geospatial Creator.)

I added CesiumGlobeAnchor to the model GameObject and set it to localToGlobeFixedMatrix based on the modelMatrix data (double array) sent from CesiumJS.
However, the model remains at the origin position.

using CesiumForUnity;
using System;
using Unity.Mathematics;
using UnityEngine;
using WebSocketSharp;

public class Sample : MonoBehaviour
{
    [Serializable]
    private class SyncMessage
    {
        public string type;
        public string fileName;
        public double[] modelMatrix;
    }

    private WebSocket ws;

    void Start()
    {
        SetupWebSocket();
    }

    private void SetupWebSocket()
    {
        ws = new WebSocket("ws://localhost:8998");
        ws.OnOpen += (s, e) =>
        {
            Debug.Log($"Connected");
        };
        ws.OnMessage += (s, e) =>
        {
            var msg = JsonUtility.FromJson<SyncMessage>(e.Data);
            if (msg.type == "add")
            {
                var go = new GameObject();
                var gltf = go.AddComponent<GLTFast.GltfAsset>();
                gltf.Url = $"http://localhost:8998/download/{msg.fileName}";
                var anchor = go.AddComponent<CesiumGlobeAnchor>();
                var mx = msg.modelMatrix;
                anchor.localToGlobeFixedMatrix = new double4x4(
                    new double4(mx[0], mx[1], mx[2], mx[3]),
                    new double4(mx[4], mx[5], mx[6], mx[7]),
                    new double4(mx[8], mx[9], mx[10], mx[11]),
                    new double4(mx[12], mx[13], mx[14], mx[15])
                );
            }
            if(msg.type == "...")
            {
                // ...
            }
        };
        ws.OnOpen += (s, e) =>
        {
            Debug.Log($"Closed");
        };
    }
}

Kevin_Ring · June 2, 2023, 12:32pm

A CesiumGlobeAnchor can only work if it’s nested inside a game object with a CesiumGeoreference. If you look in the Output Log, you should see a message to that effect. The georeference is needed to map the ECEF globe coordinates into the Unity world.

gtk2k · June 3, 2023, 6:01am

thank you.

I managed to apply the modelMatrix, but I think it’s probably due to the difference in the coordinate axes between Unity and WebGL, but it’s rotated about 90 degrees.
How can I convert modelMatrix for Unity?

Kevin_Ring · June 5, 2023, 7:13am

Ok, here’s where it gets a little tricky. The problem now probably comes down to implicit rotations applied by the two engines.

CesiumJS, by default, applies an automatic rotation to loaded glTF models in order to transform glTF’s Y-Up system to CesiumJS’s Z-Up. So if you supply a modelMatrix, the actual matrix that CesiumJS uses is modelMatrix * Y_UP_TO_Z_UP. And that should yield ECEF coordinates.

Now Unity… Unity uses a left-handed, Y-up coordinate system, unlike glTF’s right-handed coordinate system. So whatever you’re using to load glTFs in Unity (looks like GLTFast.GltfAsset?) will have to do some kind of transformation to account for that. With any luck, it’s the same one that Cesium for Unity uses to convert ECEF to Unity coordinates, and so we can effectively ignore it.

So, what I think you need to do is multiply your mx matrix times a Y_UP_TO_Z_UP matrix, and then set localToGlobeFixedMatrix with the result. You can find the Y_UP_TO_Z_UP matrix in the 3D Tiles docs here:

[
1.0, 0.0,  0.0, 0.0,
0.0, 0.0, -1.0, 0.0,
0.0, 1.0,  0.0, 0.0,
0.0, 0.0,  0.0, 1.0
]

Let me know how that goes!

chiqiangzhe · June 28, 2023, 11:12am

I added a script to the CesiumSubScene that stores the transformation matrix and iterates over all of the CesiumGlobeanchors in the update, using its internal localToGlobeFixedMatrix for assignment, *The model is not positioned correctly

void LateUpdate()
{
colliders = GetComponentsInChildren();
if (colliders != null)
{
for (int i = 0; i < colliders.Length; i++)
{
// Use this script to convert only once
if (colliders[i].gameObject.GetComponent() == null)
{
colliders[i].gameObject.AddComponent();
//if (colliders[i].transform.localScale.x == -1)
//{
// colliders[i].transform.localScale = new Vector3(-10.78795f, -5.000624f, -10.86951f);
//}
colliders[i].GetComponent().localToGlobeFixedMatrix = new double4x4(
new double4(mx[0], mx[1], mx[2], mx[3]),
new double4(mx[4], mx[5], mx[6], mx[7]),
new double4(mx[8], mx[9], mx[10], mx[11]),
new double4(mx[12], mx[13], mx[14], mx[15])
) ;
}

Seeing the article you recommended, I added the rotation matrix

colliders[i].GetComponent().localToGlobeFixedMatrix = new double4x4(
new double4(mx[0], mx[1], mx[2], mx[3]),
new double4(mx[4], mx[5], mx[6], mx[7]),
new double4(mx[8], mx[9], mx[10], mx[11]),
new double4(mx[12], mx[13], mx[14], mx[15]))*
new double4x4
(new double4(1.0, 0.0, 0.0, 0.0),
new double4(0.0, 0.0, -1.0, 0.0),
new double4(0.0, 1.0, 0.0, 0.0),
new double4(0.0, 0.0, 0.0, 1.0));

The results are still wrong

image2047×588 134 KB

[hope to get help]:

Kevin_Ring · June 28, 2023, 11:35am

I noticed in your code that you’re attempting to multiply double4x4 using the * operator. To truly everyone’s surprise, Unity has decided that multiplying two double4x4 instances in the obvious way will do a component wise multiplication, rather than a proper matrix multiplication. Strange but true!

So if you have:

double4x4 a = ...
double4x4 b = ...

You have to multiply them like this:

double4x4 result = math.mul(a, b);

If you instead do a * b, you’ll get a completely different and wrong answer.

chiqiangzhe · June 29, 2023, 3:36am

After replacing the model, I found a problem. Since the model was far away from the target position after conversion, there was no tile generation in the line of sight when running the program, so it could not be converted in the update.
Changing the value assumes that the CesiumGlobeAnchor has been generated！！
Before, I positioned the model position under edit, then roughly moved the Cesium3DTileset node to the target position manually, and then cycled the CesiumGlobeAnchor collection under the query node for scaling. This method is not universal and not very accurate. Can you give the method steps of how to use modelMatrix to locate the problem? I have tested many possibilities and still cannot solve the problem well.

Kevin_Ring · June 29, 2023, 4:02am

I’m not really following you, and I can’t write your code for you. But if you have specific questions I can try to answer them.

Are you the same person as gtk2k above, or do you work with them? That person said they had the transformation working, except it was off by 90 degrees. Simply adding in the Y-up to Z-up should fix that, so I’m a little confused about where these new and unclear problems are coming from.

gtk2k · June 30, 2023, 2:28am

I am a different person.Nor is it a colleague.
The issue has not been resolved.
Fixed by considering how to pass the values of longitude/latitude/height/heading/pitch/roll instead of modelMatrix to Unity.

Kevin_Ring · June 30, 2023, 3:25am

Glad you got it working at least. What happened when you tried to apply the extra rotation?

gtk2k · June 30, 2023, 12:04pm

My solution

/*
model.transform is the data sent from CesiumJS
{
   longitude: 0.0,
   latitude: 0.0,
   altitude: 0.0,
   heading: 0.0,
   pitch: 0.0,
   roll: 0.0,
   scaleX: 0.0,
   scaleY: 0.0,
   scaleZ: 0.0
}
*/
var jsTransform = modelInfo.transform; 
var ecef = CesiumWgs84Ellipsoid.LongitudeLatitudeHeightToEarthCenteredEarthFixed(new double3(jsTransform.longitude, jsTransform.latitude, jsTransform.altitude));
// go = GameObject
go.transform.position = (float3)geoReference.TransformEarthCenteredEarthFixedPositionToUnity(ecef);
go.transform.rotation = Quaternion.Euler((float)-jsTransform.pitch, (float)jsTransform.heading + 90f, (float)-jsTransform.roll);
go.transform.localScale = new Vector3((float)jsTransform.scaleY, (float)jsTransform.scaleZ, (float)jsTransform.scaleX);

chiqiangzhe · July 1, 2023, 12:50am

I have a model converted to a 3dtile file, loaded via Cseium 3D Tilset, but the position is very skewed (the default tile generated position is out of camera range, the run cannot load the tile)。
I want to correct the deviation with the model matrix (position, Angle and zoom values). But the premise is that the tile is in camera range and has been loaded.
Above I used the localToGlobeFixedMatrix parameter value of the CesiumGlobeAnchor on all dynamically generated tiles to be assigned to the model matrix (corrected to the Angle range if correct), but none of the tiles were generated, let alone the CesiumGlobeAnchor.

Original idea:
Cseium 3D Tilset——>CesiumGlobeAnchor——>ocalToGlobeFixedMatrix
Reality:
Cseium 3D Tilset——>not generate( The default position is out of camera range)

Kevin_Ring · July 3, 2023, 5:35am

@chiqiangzhe I think you might be overthinking this. Or perhaps I still don’t understand what you’re trying to do. But if you want to adjust a tileset by translating, rotation, or scaling it, just modify the Cesium3DTileset or CesiumGeoreference’s Transform using normal Unity methods. No need to reach into every tile and try to manipulate the transformation. Just set the Transform of the parent, and that transformation will apply to the children as well.

chiqiangzhe · July 12, 2023, 7:44am

NAME	TYPE	DEFAULT	DESCRPTION
url	Network resource/string		Tile JSON url
modelMatrix	Matrix4(Matrix4)	Matrix4.IDENTITY	The 4x4 conversion matrix of the root tile of the conversion tile set

In cesiumjs，I call Following function， and Correctly positioned my tile resources
modelMatrix: Cesium.Matrix4.fromArray([
10.307518184037022, -1.840331301212637, 2.2041437311398484, 0,
1.6249475137735327, 10.510252086834296, 1.1764976194946968, 0,
-2.3674066957807045, -0.7986124047198713, 10.404210866942293, 0,
22645464.039225746, -42494999.12734722, -38710026.09518833, 1
]
However, I did not have the localToGlobeFixedMatrix parameter in the Cesium3DTileset script in unity, so I added the Cesium3DTileset node to the CesiumGlobeAnchor. By assigning the model matrix, Did not get the same position correction as cesium js, using Y_UP_TO_Z_UP matrix, still wrong.

I actually want to do model matrix correction on the model population loaded by the Cesium 3DTileset script.
But it seems that all attempts are not enough for me to achieve the same effect as the Cesiumjs
In the figure above, since positon has a large offset value relative to vector(0,0,0), scaling is carried out according to the center point, resulting in changing the scale value also changing its position relative to vector

Kevin_Ring · July 12, 2023, 10:25am

I’m having trouble following you. But I think maybe you’re trying to position a tileset that isn’t georeferenced? In other words, it has a local origin, rather than an origin at the center of the Earth.

If I have that right, the trick is to create a second CesiumGeoreference with the origin set to the center of the Earth (ECEF 0,0,0) and then put your tileset as a child of that one. It sounds a little counter-intuitive, but that will effectively disable the georeferencing, allowing your tileset to be transformed in the Unity world just like any other Game Object.

chiqiangzhe · July 12, 2023, 2:50pm

Yes, my longitude and latitude values default to the position of the longitude and latitude values in the center of my country’s capital, so the backend provides a transformation matrix to correct the error. Will two Cesium Georeferences cause performance degradation for my tile set under subscription? I will try the method you mentioned as soon as possible. Thank you

chiqiangzhe · July 13, 2023, 3:03am

It’s been tested and it doesn’t work very well,Now it can only be dragged by hand, the node pivot is too far away from the production tile, and rotation and scaling will make the model position change a lot

Kevin_Ring · July 13, 2023, 9:40am

Will two Cesium Georeferences cause performance degradation for my tile set under subscription?

No.

Now it can only be dragged by hand, the node pivot is too far away from the production tile, and rotation and scaling will make the model position change a lot

Ok, that’s probably because the tileset is georeferenced, but incorrectly. My suggestion is only applicable if the tileset has a local origin near its center.

This is tricky to get right because CesiumJS and Unity have very different coordinate systems. CesiumJS uses ECEF. Unity uses… a local, left-handed coordinate system defined by the CesiumGeoreference. To use the CesiumJS matrix you’ve already determined, you would need to multiply it into the transformation chain between the “ECEF to Unity” transformation and the “tile to ECEF” transformation. There’s not really any way to do that.

So the next best thing is to compose a transformation like this, and assign it as the Unity game object Transform:

[ECEF to Unity] * [Your Transformation from CesiumJS] * [Unity to ECEF]

You can get [ECEF to Unity] from the CesiumGeoreference as the ecefToLocalMatrix property. [Unity to ECEF] is localToEcefMatrix. Multiply them all together, in that order. Don’t forget that you must use Math.mul to multiply double4x4 matrices; the operator* does a component-wise multiplication instead of a proper matrix multiplication.

Or maybe it’s easier to just eyeball it with the Unity transform widget, rather than trying to use the matrix from CesiumJS?