Questions about usage of EXT_structural_metadata gLTF extension

CesiumJS
1

Hi all,

I am currently working on creating 3DTiles 1.1 tilesets (with GLTF files) for visualization in CesiumJS.
To add arbitrary metadata to the tiles, I am using the EXT_structural_metadata extension together with the EXT_mesh_features extension.

I have some questions about the usage of the concepts of the EXT_structural_metadata extension and its current implementation in CesiumJS.
As the concepts are still quite new and therefore not completely implemented in CesiumJS yet, it’s sometimes a bit difficult to see if I misunderstand the specifications or the capabilities of Cesium, so please bare with me. :smile:

(As this is mainly about the usage of those concepts in CesiumJS I added the question to the CesiumJS category, although it overlaps with the 3DTiles category.)

  1. I am currently using the same buffer views in the gLTF for PropertyAttributes and for a PropertyTable. The PropertyTable is using the buffer views directly to reference the property data. The PropertyAttributes use accessors on the same buffer views.
    The data in the buffer is aligned to 4 bytes (every value is either 4 bytes big or padded with zeros to equal 4 bytes) and the buffer views define a byte stride of 4.
    Now in CesiumJS it looks like the data is correctly read from the PropertyAttributes (in a custom shader) but when accessed via the PropertyTable (on the “cpu side” in the viewer) the data is incorrect.
    Should my approach work or does the property table need a different alignment or something similar? I am retreiving the data from the PropertyTable (of a picked tileset) via content.batchTable.getProperty().
    It looks like some values are correct but others return the same value as a neighboring primitive (which makes me believe there is some “bleeding” happening from the range of one value into the other one).
    If I use a tileset with only one single primitive (and only one property) then the value of that property is correct in the PropertyAttribute and the PropertyTable.
  2. Is it correct that PropertyAttributes can’t contain integers bigger than 16 bit (because that’s part of the gLTF specification for accessors) while PropertyTextures and PropertyTables can contain bigger integers (as they don’t use accessors)?
  3. If so, is it correct that it is currently not possible to use integers > 16bit for visualization purposes in CesiumJS custom shaders (because PropertyTables are not available and CesiumJS currently only supports UINT8 values in PropertyTextures)?
  4. Is it correct that by using PropertyAttributes you are limited to a relatively small number of attributes because they are added as vertex attributes and most GPUs only support the usage of 16 vertex attributes which also includes texture coordinates etc.?

Thank you!

2

This is a first quick “triage” of the questions. There may be some details that we have to zoom in, but maybe a few points can already be sorted out quickly in this first pass:

I am currently using the same buffer views in the gLTF for PropertyAttributes and for a PropertyTable. […]
Now in CesiumJS it looks like the data is correctly read from the PropertyAttributes (in a custom shader) but when accessed via the PropertyTable (on the “cpu side” in the viewer) the data is incorrect.
Should my approach work or does the property table need a different alignment or something similar?

I’ll have to look into the implementation and specification, and maybe try to create “similar” data. (If you could share one of these files, and maybe a Sandcastle that tries to read the data, then this could probably be done much more quickly).

There are alignment requirements for the binary data as a whole (mentioned here), but beyond that, the only requirement is that the byteOffset of each buffer view must be aligned to a multiple of its component size, which is in line with the glTF specification (but also applied to 64bit values here).

One part sounds suspicious, though:

(every value is either 4 bytes big or padded with zeros to equal 4 bytes)

When you have, for example, a set of 16 bit values (a, b, c…) then the layout will be

[ a0 a1 b0 b1 c0 c1 ...]
and not
[ a0 a1 __ __ b0 b1 __ __ c0 c1 __ __ ...]

Just to emphasize that the alignment does not apply to indvidual values here, but to the data block as a whole.

Is it correct that PropertyAttributes can’t contain integers bigger than 16 bit (because that’s part of the gLTF specification for accessors) while PropertyTextures and PropertyTables can contain bigger integers (as they don’t use accessors)?

Yes. This is mentioned in the specification as

The property types that are supported via property attributes are therefore restricted to the types that are supported by standard glTF accessors.

This was one of the limitations that we had to accept, inheriting it from the glTF specification. (I think that this is really unfortunate - similar to the 16-bit-indices limitation of glTF. Maybe this can be lifted via extensions in the future. There might be ways to “cheat” around that limitation, but none that I would recommend…)

If so, is it correct that it is currently not possible to use integers > 16bit for visualization purposes in CesiumJS custom shaders (because PropertyTables are not available and CesiumJS currently only supports UINT8 values in PropertyTextures)?

I think that this is correct. The details about what is supported to which extent are tracked in issues like 3D Tiles Next Metadata Compatibility Matrix · Issue #10480 · CesiumGS/cesium · GitHub , but that might have to be reviewed and updated at some point.

Is it correct that by using PropertyAttributes you are limited to a relatively small number of attributes because they are added as vertex attributes and most GPUs only support the usage of 16 vertex attributes which also includes texture coordinates etc.?

The texture coordinates are usually float (32 bit), so I’m not sure what this refers to.

But when the goal is to store integer values ((u)int8/16), then the number of possible attribute values is limited to 256/65536 values. (I assume that this is what you referred to by “number of attributes”)

4

Hi Marco,

thanks for the quick reply!
Usually we always try to provide a sandcastle if we have a problem, however in this case it’s a bit hard at this stage, as I am currently juggling around with different prototypes of my converter.
However, I will try to produce a gLTF that has a minimal amount of data in it and extract the relevant code I use to load it.

I think your answer already touches some points, that help clarify things up a bit!

Just to emphasize that the alignment does not apply to indvidual values here, but to the data block as a whole.

This actually was what I intially also thought and worked with. However, I couldn’t get it to work properly and recently switched to adding a padding to the single values. You understood that correctly.
I based this of this older post in the gLTF repo: Clarification on byteStride requirement · Issue #1198 · KhronosGroup/glTF · GitHub
which says

When the buffer view is used by just one accessor, it’s somewhat reasonable to assume that the effective stride is equal to the componentLength * componentCount.
For instance, FLOAT Vector3 accessors have a stride of 4 * 3 = 12.
So the spec allows omitting it in such a case unless the computed stride is not divisible by 4 (GPU requirement), for instance, Vector3 with UNSIGNED_SHORT components must have a stride at least of 8 bytes instead of 3 * 2 = 6,i.e., it has to be stored as x|x|y|y|z|z|0|0.

So my thought was if the stride works like this, it should also work if I pad every single value to 4 bytes. :wink:
However, I already thought that this seems a bit wasteful, so I will have a look at this again.
Still, shouldn’t this work nonetheless? As long as the byte stride is set to the correct value, shouldn’t paddding 0s be ignored?
As I said, I am pretty sure that the values are correct on the GPU (when read as PropertyAttributes).

I think that this is really unfortunate - similar to the 16-bit-indices limitation of glTF

Okay, then I understood that correctly and also understand now that I am not missing something but that the gLTF spec is just not a good fit for some 3DTiles use cases here. :slight_smile:

The details about what is supported to which extent are tracked in issues like …

Yes, I think it was you who pointed me to that issue before! ^^
Thanks again, this time I checked it beforehand. I just wanted to make sure that I am not missing anything else.

The texture coordinates are usually float (32 bit)

Maybe this was a bad example. What I am referring to is the limit of 16 vertex attributes per vertex that most GPUs have.
Maybe texture coordinates are not part of the vertex attributes, I am not sure.
However, If I am not mistaking then PropertyAttributes are sent to the GPU as vertex attributes. I think this means that the hardware limit of 16 vertex attributes will limit the usage of PropertyAttributes to max. 16 attributes.
I already tried to add more than 16 attributes and actually got an error like “trying to bind too many vertex attributes”.

Feel free to add some more insights if something comes to your mind. :slight_smile:
Otherwise I will come back to this thread with an example of one of my gLTF as soon as I managed to produce something meaningful.
Or (would be even better) come back with a solution if find my error!

Best regards,
Marcel

5

This actually was what I intially also thought and worked with. However, I couldn’t get it to work properly and recently switched to adding a padding to the single values. You understood that correctly.
I based this of this older post in the gLTF repo: …

The reference here is the glTF specifiction, and 3.6.2.4 Data Alignment says

For performance and compatibility reasons, each element of a vertex attribute MUST be aligned to 4-byte boundaries inside a bufferView

So this applies to property attributes as well (because they are vertex attributes).

But the Metadata specification says

Property values are binary-encoded according to their data type, in little-endian format. Values are tightly packed: there is no padding between values.

So there is no requirement for padding of the values of property tables.

Now, re-reading your initial post in light of that, it might be that I just fully understood the statement

I am currently using the same buffer views in the gLTF for PropertyAttributes and for a PropertyTable . […]

Does this mean that they both refer to the same region as well? So you are storing, for example, 3D float vectors in a buffer view like

[ 0.0 1.0 2.0 ... ]

and want to use the identical memory region once for a property attribute, and once for a property table?

If this is your goal: I’m not sure whether we ever even considered this while writing the specification. But it could raise a lot of questions - and the alignment is only one of them. Maybe that’s also the reason for the next question:

Still, shouldn’t this work nonetheless? As long as the byte stride is set to the correct value, shouldn’t paddding 0s be ignored?

That may be the crucial point here: I think that the CesiumJS code for accessing the buffer views of property tables should not take the byteStride into account (and I cannot imagine that it does, but would have to verify that).

Generally, for data types that do not have 4 bytes, you could not find an alignment that fulfulls the requirement of glTF (4-byte-aligned) and of the Metadata specification (being tightly packed).

(For the 4-byte-case, it might even be “technically valid”, but … it would be unusual, at least, and I’d have to think this through more thoroughly…)

Therefore, on a higher level: Why would you want to fill a property table with a column that stores exactly the value of a vertex attribute in each row?

The texture coordinates are usually float (32 bit)

Maybe this was a bad example. What I am referring to is the limit of 16 vertex attributes per vertex that most GPUs have.

It wasn’t a bad example. I just misunderstood the point. Indeed, the GPU will very likely have a limit for the number of vertex attributes. And depending on what that limit is (and which other vertex attributes are used), this will limit the number of possible metadata attributes. For example, if the GPU has a limit of 16, and are aleady using position, normal, tangent, bitangent, and maybe even multiple sets of texture coordinates, then you might hit this limit when trying to add many property attributes.

I think this also sounds like an unusual case, but may be worth examining as one “stress test”. There currently is no strict requirement on the specification level, about how many property attributes are considered to be “valid” or “required to be supported by clients”.
If this comes up in practice, there might be workarounds (like … not storing the values themself, but only IDs via EXT_mesh_features, and use these to look up the data in a property table).

Otherwise I will come back to this thread with an example of one of my gLTF as soon as I managed to produce something meaningful.

That would be great.
(I could also try to create a few tests, like that of “identical buffer view regions for property attributes and property tables”, or that of “many (>16) attributes”, but this may more sensibly be tackled as a dedicated task in the context of more systematically creating models that try to find limits and corner cases of the specification…)

6

Oh this makes way more sense now!
I knew I read somewhere something about values being tightly packed, but didn’t find it again.
I think I struggled mentally juggling the different specifications for gLTF, 3DTiles and the gLTF extensions. :slight_smile:

I will try to create a second buffer for the property table. This was my intended way initially, but I stopped that path (as I didn’t know/forgot about the different alignment rules) because it seemed wasteful (memory usage wise).
For now let me clarify the purpose of my question. I see now that my ideas seem a bit odd, if you don’t know the backgrounds.

  1. I have vector data (e.g. shapefiles) and want to display them in CesiumJS.
  2. They can be too big to load them all at once, so tiling is needed.
  3. I want to use 3DTiles for that as we already use the format for other purposes and it seems to be the only more or less fitting option inside CesiumJS for the use case.
  4. 3DTiles 1.1 does not really support vector data (and 3DTiles 1.0 only includes the never-completed VCTR standard).
  5. That’s why I am converting the vector data into gLTF to add the data to a 3DTiles tileset.

This is the reason why I deal with a bigger number of properties than one may might expect when creating 3D tilesets.
Vector data can contain an arbitrary amount of properties. As a made up example consider a shapefile that contains one polygon for every building in a city and these polygons contain many properties about that building (e.g. area, height, owners, residents, …).

I now want to

  1. show that information to the user (e.g. if they click a polygon in the viewer)
  2. color the polygons based on one of the properties in a custom shader (e.g. all polygons with more than 5 residents red, rest white)

This creates some problems in the current implementation of EXT_structural_metadata in CesiumJS:

  1. According to the matrix you linked above, only property textures and property attributes are usable in custom shaders. Property textures only support Uint8, so only property attributes are usable for me.
  2. According to the same matrix, I can’t use property attributes for picking (and therefore showing the values in a list in the viewer).
  3. Additionally, as mentioned above, it is easily possible in my use case to hit the hardware limit for vertex attributes if I’d add all of the properties as property attributes. So even if I could show their values in the viewer, I would only be able to store the first 10 attributes or so.
  4. Picking is only available for property tables and they are also capable of storing any number of properties. That’s why in addition to storing some properties as property attributes (for usage in the shader) I also want to store all properties in a property table (for showing their values in the viewer).

Based on this line of thought I did some experiments and ended up with the idea to use the same buffer for property attributes and a property table. This way I wanted to avoid to store the same data twice in the gLTF.
I see now that - besides the alignment - this idea might only be good in theory, because this is not something the format encourages you to do and therefore there’s probably no guarantee that this will always work.
I also just thought about another difference:
The property attributes buffer needs one value per vertex. The property table buffer probably only needs one value per primitive (in my case)? Is that correct?
I remember that I wasn’t sure about that and wanted to try that out (but then hit the alignment problems first).

I hope this sheds some light on the strange things that are happening in my code.

I just realized that until now I completely ignored the Styling column in the compatibility matrix you linked.
As property tables can be used for picking and for styling, a better workaround for the whole problem might be to not use custom shaders in this case but fallback to the styling language, I guess?
We already style other 3DTiles in our code and completely moved away from using the styling language once custom shaders were available. The shaders are just more flexible and the styling language is just an additional layer between our code and the final shader anyways. :slight_smile:
Still, right now I think, using the styling language just for the current use case might be the most viable option for us, although we’d have to rewrite that part of the code once again if property tables become available in custom shaders (if ever :slight_smile:).

For the sake of the experiment I will try to create a working gLTF using two buffers (one for the property attributes and one for the property table). If I want to use the property tables for styling, I have to make them work anyways!
In case the styling language fulfills our needs for now, I will remove the property attribute part from my code.
Nonetheless you were a great help here and I am 95% sure that you solved my problem! :smiley:

So I am not sure if I could provide you a meaningful example gLTF when I change the code to use two different buffers.
As you said, if you want to properly test more edge cases, you might have to create your own fitting models anyways.
However, if you have any idea of a gLTF that I can provide that can help you in any way, let me know. :slight_smile:

7

I hope this sheds some light on the strange things that are happening in my code.

Indeed, it does. But there certainly are many elements that are specific for the task and the application, and where solid recommendations would require a much deeper knowledge about things like Shapefiles and vector data than I have. The general demand for (better) vector data support is on the radar, but … admittedly, Vector Tiles · Issue #2132 · CesiumGS/cesium · GitHub has been open for a while now, and it’s hard to summarize the latest state.

This is the reason why I deal with a bigger number of properties than one may might expect when creating 3D tilesets.

Vector data can contain an arbitrary amount of properties. …

I now want to …

This creates some problems in the current implementation of EXT_structural_metadata in CesiumJS:

Unfortunately, I cannot provide a blanket solution for each of the requirements or workarounds for the (current) limitations of CesiumJS. I thought that the approach of using EXT_mesh_features for assigning IDs, and looking up data for these IDs in EXT_structural_metadata property tables should be a versatile solution, but there probably are limitations (for picking or custom shading) that are not covered by this.

If there is a specific functionality, then it could make sense to try this carve out into a very specific, narrow question in a new thread. Maybe someone else can give hints or better recommendations here. (To be honest: Nobody else is going to read these walls of text that we are creating here :smiley: )

The property attributes buffer needs one value per vertex. The property table buffer probably only needs one value per primitive (in my case)? Is that correct?

If I understood this correctly, then this is depending on the required granularity. When you have a mesh primitive with 100 vertices, and all vertices have “the same metadata value”, then it probably wouldn’t make sense to store this value, in a property attribute, 100 times. It could be one row in a property table. But … if only 97 vertices have value A, and 3 vertics have value B, then … this information has to be stored somewhere. And very abstractly, there could be two options:

  • Directly storing the different values in the property attribute - 99 times value A, and 3 times value B
  • Assigning IDs to these vertices - namely, 97 times the ID "0", and 3 times the ID "1" - which can be used to look up the metadata in ‘row 0’ and ‘row 1’ of one property table, respectively

The approach of using an ID obviously is a sort of “detour”. Instead of directly looking up
vertexIndex --[propertyAttribute]--> metadataValue
the path would be
vertexIndex --[propertyAttribute]--> ID --[propertyTable]--> metadataValue

Both have pros and cons:

  • When the metadata value is “large” (like a long STRING or VEC3/array), then the latter would be more space efficient: Instead of storing it 99/3 times, it would only be stored 1/1 times (once in each row of the 2-row-property table)
  • but this detour might cause other limitations (e.g. not being able to use the metadata value in custom shaders or so…)

This is related to your next point:

I just realized that until now I completely ignored the Styling column in the compatibility matrix you linked.

It might, in some cases, be possible to work around certain limitations of metadata access in custom shaders, by doing the styling “manually, on the CPU side”. Again, sorry for remaining so vague here. For now, I can only look at that matrix and try out specific examples (for example, of things that should work, but don’t), but cannot give profound recommendations about which approach may be the best (best supported or best suited) for which application case.
Carving out the specific case into a dedicated thread might help here.

However, if you have any idea of a gLTF that I can provide that can help you in any way, let me know. :slight_smile:

That request mainly referred to the glTF that you mentioned in the first post, and where you said

Now in CesiumJS it looks like the data is correctly read from the PropertyAttributes (in a custom shader) but when accessed via the PropertyTable (on the “cpu side” in the viewer) the data is incorrect.

In hindsight, the “shared buffer region” may explain that. Beyond that: Creating dedicated test models for this case could be done, either as actual .gltf files or programmatically, within the CesiumJS specs/unit tests.

8

Thanks for another extensive answer!

I think this is all the info I need for now. I know that my question (especially initially) was a bit vague and more about the concepts rather than the actual use case.
So I am more than happy with your answers!

I see the whole thing much clearer now and I think I found a path that should work.
I can already confirm that using different buffers for PropertyAttributes and PropertyTable solves my initial problem with having wrong values in the table.

I will now try to omit the PropertyAttributes and only use the PropertyTable and try to style the tileset based on the table.

As you said, in case I have any more specific questions, I’ll open a new thread. :slight_smile: