Gltf embed metadata, EXT_mesh_features, EXT_structural_metadata

3D Tiles
1

I am working on the custom tileset.json, and I have the valid gltf files now, I want to embed the metadata to the corresponding gltf files.

Description: I am working on the powerlines network, there are many of the powerlines more than 10,000 of them. A powerline is a curved line. I want to embed the metadata to each of the gltf, so I could use the 3D style condition to dynamically apply style to it like this:

const voltageStyle = new Cesium.Cesium3DTileStyle({
    color: {
        conditions: [
            ["${Voltage} === 240", "color('yellow')"], // Show tiles where Voltage is 240 or higher
            ["${Voltage} > 0", "color('red')"], // Show tiles where Voltage is 240 or higher
            ["true", "color('blue')"], // Hide all other tiles
        ],
    },
});

and here is the sample sandcastle example to describe the powerline loaded as tileset, I am still updating my latest one on sandcastle.

Task: The expected feature for now is:

  1. style with different color based on voltage
  2. mouse event to show info box for its corresponding data for each single powerline

Method tried: I have tried the tileset metadata method MetadataGranularities, but a grouped tileset can only apply one style to that tileset, I mean it couldn’t change a single part style inside that tileset. I successfully used this method to show corresponding metadata to the parts of the grouped tileset.

Difficulties/issue:
I had multiple tries with the gltf but still cannot figure out how to use extensions, and I followed EXT_structural_metadata/FeatureIdAttributeAndPropertyTable this when I tried. Here is one of my gltf file, it is a powerline like a curved line made of 11 cartesian points, the coordinate system is using Cesium when converting. When I created those powerline, it is grouped based on its tiles coordinate.

To be more specific, I think my issue would be assigning feature id and assigning feature id to property table.

{
    "asset": {
        "version": "2.0"
    },
    "scenes": [
        {
            "nodes": [0]
        }
    ],
    "nodes": [
        {
            "matrix": [1, 0, 0, 0, 0, 0, -1, 0, 0, 1, 0, 0, 0, 0, 0, 1],
            "children": [1]
        },
        {
            "mesh": 0
        }
    ],
    "meshes": [
        {
            "primitives": [
                {
                    "mode": 3,
                    "attributes": {
                        "POSITION": 0
                    }
                }
            ]
        }
    ],
    "buffers": [
        {
            "uri": "data:application/octet-stream;base64,JLKZwnlUUkI2QO3C0EebwuMlU0IR4erC6PmcwskbVEJIoejCPciewu81VUKlgObCvbKgwj50VkIVf+TCcrmiwsLWV0KinOLCg9ykwq1dWUJ42eDCMRynwlQJW0LgNd/C33ipwjPaXEJEst3CCPOrwuzQXkIuT9zCR4uuwkTuYEJKDdvC",
            "byteLength": 132
        }
    ],
    "bufferViews": [
        {
            "buffer": 0,
            "byteOffset": 0,
            "byteLength": 132,
            "target": 34962
        }
    ],
    "accessors": [
        {
            "bufferView": 0,
            "byteOffset": 0,
            "componentType": 5126,
            "count": 11,
            "type": "VEC3",
            "max": [-76.8479318455793, 56.23268031002954, -109.52595210261643],
            "min": [-87.27202824316919, 52.58249282790348, -118.62541104014963]
        }
    ]
}

here is the data to this gltf

[
    {
        "Bay_Id": "173840",
        "ConductorId": "514490",
        "Conductor_Length": "14.537233",
        "Coordinates": [
            {
                "longitude": 2.570975183209337,
                "latitude": -0.7482747440108009,
                "height": 148.9212520894381
            },
            {
                "longitude": 2.570975237863152,
                "latitude": -0.7482745243966467,
                "height": 148.68407230428997
            },
            {
                "longitude": 2.570975292516946,
                "latitude": -0.7482743047824905,
                "height": 148.53690032648962
            },
            {
                "longitude": 2.5709753471707173,
                "latitude": -0.7482740851683327,
                "height": 148.47915607742527
            },
            {
                "longitude": 2.570975401824467,
                "latitude": -0.7482738655541729,
                "height": 148.51061195803672
            },
            {
                "longitude": 2.570975456478194,
                "latitude": -0.748273645940011,
                "height": 148.63139195173324
            },
            {
                "longitude": 2.5709755111318997,
                "latitude": -0.7482734263258474,
                "height": 148.84197211307438
            },
            {
                "longitude": 2.5709755657855835,
                "latitude": -0.7482732067116817,
                "height": 149.14318244413997
            },
            {
                "longitude": 2.5709756204392455,
                "latitude": -0.7482729870975143,
                "height": 149.53621016598464
            },
            {
                "longitude": 2.5709756750928854,
                "latitude": -0.7482727674833449,
                "height": 150.02260439807182
            },
            {
                "longitude": 2.570975729746503,
                "latitude": -0.7482725478691736,
                "height": 150.6042822641308
            }
        ],
        "Voltage": "1245.0",
        "Tile": { "x": 119168, "y": 48377, "level": 16 }
    }
]

and here is the tileset.json that I loaded to the scene

{
  "asset": {
    "version": "1.1"
  },
  "geometricError": 4096,
  "root": {
    "boundingVolume": {
      "box": [
        -82.05997848510742,
        114.07568359375,
        54.4075870513916,
        5.212047576904297,
        0,
        0,
        0,
        -4.5497283935546875,
        0,
        0,
        0,
        1.825094223022461
      ]
    },
    "geometricError": 512,
    "content": {
      "uri": "514490.glb"
    },
    "refine": "ADD"
  }
}

please leave any ideas to my question or leave a correct gltf if it is possible, many thanks. If need more info regarding to this, please leave any comments as well.

2

There are many degrees of freedom for structuring data sets in general. And the point of metadata adds a whole new dimension to these existing degrees of freedom. The question about what is “The Right Structure” is an engineering question. It is impossible to give profound recommendations here, without a diligent and detailed analysis of the requirements and goals.

For each possible solution, there are pros and cons, or possible caveats and constraints. As mentioned in an earlier thread, where you asked about this, one constraint is that there currently appears to be a bug in CesiumJS where styling based on metadata is not applied.

Therefore, the following should be considered as a “workaround”. It is not ‘the recommended solution’. It only shows one way of how it might be possible to achieve what you want, involving many, many guesses and assumptions.

I took the glTF file that you provided, and created two additional copies of that (moved about 10 and 20 in x-direction, just to have them at different positions). I created a tileset.json that refers to these glTF files. (And this uses a proper bounding box - the one that you provided did not match the data).

In that tileset, I defined a metadata schema - roughly corresponding to the metadata that you provided. (I only omitted the ‘tile’ information. You’ll have to think about how to encode that sensibly. The structure of "Tile": { "x": 119168, "y": 48377, "level": 16 } does not directly match one of the metadata types, but you could store it as a VEC3/INT32 or so - similar to how I stored the coordinates as a VEC3/FLOAT32 for this example)

The tileset refers to each of the GLB files. Each GLB file is one ‘tile’ in the tileset. Each of these tiles has metadata that matches the schema.

The only difference in these tile metadata entities is the Voltage property: It’s 123, 1234, and 12345 for the different tiles.

The result is in this archive:

Forum-29582-example.zip (5.3 KB)

The archive contains a Sandcastle.js that shows how this metadata might be used. The sandcastle inspects the tiles that are loaded, looks at the metadata of these tiles, and applies different styles to the content of these tiles, depending on the Voltage value:

  • For Voltage < 1000, the content is green
  • For Voltage > 1000, the content is yellow
  • For Voltage > 10000, the content is red

Again: This is a workaround for the styling bug linked above. When you say that you have 10000 tiles/powerlines, then this may cause some performance issues. It might be possible to find a better solution here, but it is intended as one example/demo.

The following is a short screencapture of that sandcastle: The power lines are rendered with different colors. The tooltip shows why - namely, because they have different Voltage values.

Cesium Forum 29582

Once more: This is not “THE” recommended solution. Whether or not this suits your needs is hard to tell. But it shows one possible approach for assigning metadata to tiles, and using that metadata for styling and inspection.

3

Thanks for the reply, I thought it is a different topic using the gltf extensions, that’s why I create another one. Yes I am trying your suggested approach, and I am processing the gltf to tileset.json

But again thank you so much for the help, without your help, I was stuck at this stage in the last two month, and this workaround seems to be ok, I will keep trying this approach and also the gltf extension, and will update you if got progress.

4

As I said in the previous thread (and in the first paragraph here): The question of what the metadata is associated with largely depends on the intended granularity.

When you have one glTF asset that represents “the powerline”, and that is directly used as the tile content, then you can associate the metadata with the tile directly, and don’t need the glTF metadata extensions.

However, when you have a complex glTF that contains multiple components, and when you even want to associate different metadata entities with individual vertices or texels within the glTF data, then you can use the glTF metadata extensions.

5

Hi, here is my work following your suggestions. data embed to tileset example, thanks for the help.

It is working good, and to use more of the gltf features I got a new task which is to use gltf to embed the data. So I am back to the issue with EXT_structure_metadata and EXT_mesh_features.

here is the example file I am working on. A gltf contains three conductors, which are generated from the data file. json file for the tileset, and a glb file.
The gltf’s buffer part uri is for the three conductors, I haven’t put the metadata to buffer yet, because I don’t know what structure it is for the data.
119227.zip (4.7 KB)

My question is:

  1. How to put metadata to the gltf buffer and use properties value point to it. (I already have the schema, and I assume the properties value points to bufferView, and data stored at buffer)
  2. How the featureId points to the property table, how does cesium know which values assign to which mesh.(my gltf have distinct mesh, each mesh primitives represent a conductor)

I have been browsing the doc regarding to this, here is the reference:

and here is how I generate for the buffer for a single conductor using its cartesian coords

export function floatsToBase64(floats) {
    // 1. Place the floats into a Float32Array
    const floatArray = new Float32Array(floats);

    // 2. Convert the buffer of the Float32Array into a Uint8Array
    const uint8Array = new Uint8Array(floatArray.buffer);

    // 3. Convert the Uint8Array to a Base64 string
    let base64 = '';
    const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';

    for (let i = 0; i < uint8Array.length; i += 3) {
        const a = uint8Array[i];
        const b = uint8Array[i + 1];
        const c = uint8Array[i + 2];
        const index1 = a >> 2;
        const index2 = ((a & 3) << 4) | (b >> 4);
        const index3 = isNaN(b) ? 64 : ((b & 15) << 2) | (c >> 6);
        const index4 = isNaN(b) || isNaN(c) ? 64 : c & 63;

        base64 += chars[index1] + chars[index2] + chars[index3] + chars[index4];
    }

    // Handle padding if necessary
    const paddingLength = 4 - (base64.length % 4);
    if (paddingLength !== 4) {
        base64 += '='.repeat(paddingLength);
    }

    return base64;
}

Any advices are welcome and appreciated. Thanks

6

I’m not exactly sure what the intention is behind this buffer creation code. The question of how you can bring the data for these extensions into the glTF file mainly depends on how you are generating the glTF file in the first place. You will need a glTF reader/writer that supports these extensions (you certainly don’t want to assemble the binary data and JSON manually, particularly when the binary data contains arrays).

There is an implementation of these extensions, based on glTF-Transform, in the 3d-tiles-tools. This is not really “public API”, but there is a demo/test for this functionality in ExtStructuralMetadataPropertyTableDemo.ts (which is part of this state at the time of writing this. Right now, this is the same as the main state, but might change).

It creates a glTF asset with EXT_structural_metadata. There is a similar demo for EXT_mesh_features in the same directory. With this functionality, it should be possible to create glTF assets that contain both. And how this data can be displayed in CesiumJS is shown in the 3D Tiles Samples, specifically, in the Sandcastle that is linked from there.

7

Thanks for the help, just update my progress. I have worked out how to use both extensions.

8

Hey all. I am working on similar problem and wanted your input on metadata extensions and gltf manipulation via gltf-transform library. Here is my use case:

1.We have IFC models of facilities, that contains hundreds of assets (valves, pipes, tanks etc).
2. We’re extracting the bounding box of every asset within the model and storing it within the database.
3. In addition to IFC models, we also have lidar photogrammetric meshes built with reality capture.
4. We’re exporting these meshes as GLB.

image
image726×600 73.3 KB

5. The structure of the GLTF:

  • 1 node
  • 1 mesh
  • 12 primitives ( bounding boxes within the image above)
  1. We want to embed the extracted asset hierarchy (with metadata) of IFC model into mesh gltf model.
  2. Finally we want to convert the glb model to 3d tiles format (Cesium Ion) so that it could be displayed in CesiumJS.

Questions:

  1. With potentially thousands of assets, is it okay to use a single node with multiple primitives instead of separate nodes for each asset?
  2. Do we need to remove the existing default primitives? Or we just append the new ones?
  3. Which metadata storage implementation would make most sense here? Property tables, property attributes…?
  4. During 3D Tiles conversion, will Cesium-Ion preserve the metadata within the GLTF? Considering it’s splitting the glb into smaller tiles.

I hope that this workflow and questions makes sense, if not I apologize for my ignorance.

9

Hey @kneitukas.

Maybe start by looking at EXT_mesh_features which will let you tag portions of the mesh and associate them with metadata. Feature IDs can be per-vertex or per-texel.

Then use a property table to store the metadata.

We’re still thinking about how to do hierarchical metadata. In the past we had an extension called batch table hierarchy but we don’t have a glTF equivalent yet. It’s top of mind with the upcoming design tiler and AEC tech preview.

Currently Cesium ion doesn’t preserve metadata for photogrammetric models, but we’re actively working on that as well.

I think it’s really cool that you’re combining IFC with scanned data. :grinning:

10

Hey @sean_lilley, thank you for you answer.

So if cesium-ion doesn’t preserve metadata yet, what would be my options?

  • Render .glb directly, not as a 3D tiles?
  • Embed the metadata within tileset.json instead of gltf?
  • Or maybe have a seperate .glb / node for every asset within the mesh, and construct tileset.json with them?

What do you think? Thank you for your help Sean.

11
  • Render .glb directly, not as a 3D tiles?

This should work in the short term, as long as your model isn’t too big.

12

It is big. 10-100 GB unfortunately :confused:

13

You cannot have a 10 GB GLB file anyhow. GLB is inherently limited to 4 GB on the specification level (but most libraries will not be able to process models that are larger than 2 GB).

So the last option that you mentioned…

… may be the most viable here. How exactly to accomplish that may depend on some details of the structure of the model and its associated metadata. For example: When you have to split the model into many smaller GLBs anyhow, you might even consider to store the metadata in the tileset JSON (when you don’t really need the fine-grained structures that are offered with the glTF metadata extensions).

There may be a few useful “building blocks” for what you are trying to accomplish in the 3d-tiles-tools. There are some demos showing the metadata extensions, to be used with glTF-Transform. And there is a createTilesetJson function that can create a tileset JSON file from multiple input GLB files. But where and how this could be assembled to suit your needs depends on the exact input, requirements, and desired output.