Intersection with glTF model

CesiumJS
1

1. A concise explanation of the problem you're experiencing.
I would like to know if it is possible to test for an intersection between a ray and a glTF model. Cesium.IntersectionTests seems to provide tests for primitive types like triangles, spheres, etc. but not for higher level types like a glTF model/entity.

2. A minimal code example. If you've found a bug, this helps us reproduce and repair it.
It would be great to be able to do something similar to this:

const viewer = new Cesium.Viewer('container', {});
const asset = viewer.entities.add({
  model: {
    uri: url
  },
});

const origin = new Cesium.Cartesian3();
const direction = new Cesium.Cartesian3();
const ray = new Cesium.Ray(origin, direction)
const intersection = Cesium.IntersectionTests.rayModel(ray, asset);

3. Context. Why do you need to do this? We might know a better way to accomplish your goal.
I am trying to project a point onto a glTF asset. You can think of the point as a camera position with a direction that is looking at a glTF model. I'd like to project this point along a ray in the direction it is facing so that I can visualize a marker on the asset at the point where the ray intersects with the model.

4. The Cesium version you're using, your operating system and browser.
Cesium: 1.39.0
OS: Ubuntu: 16.04
Browser: Chrome 62

Thank you,
Eric

2

Hi, just wanted to bump this. Any thoughts you have at all would be much appreciated. Thank you!

3

This would be a very cool feature to have. Cesium doesn’t have direct support for this type of intersection test.

There are a some possible approach here that would require a bit of work:

  1. Store a model’s triangle list on the CPU and perform ray-triangle intersections. Build an octree or other acceleration structure to speed up the checks.

  2. Utilize the ShadowMap class in some way. Create a 1x1 shadow map from an orthographic camera pointing along the ray direction and then render. Call readPixels to get the rendered depth and then project back to world space in order to place the marker.

4

Sean,

Thank you for this reply. I am curious, would either of the two approaches you listed be possible for testing intersection with rendered 3D Tiles? Or would this require a different methodology?

It seems like once you have a triangle list from anything rendered in the scene, the ray-triangle test could be performed. Obtaining the triangle list could be application specific (not sure), but after that it would be ubiquitous. Similarly it seems like the ShadowMap approach could be reused.

Thank you for your help with this.

Eric

5

Yup, either of the two approaches would work with 3D Tiles since the b3dm format uses Model under the hood. For the triangle list approach there would just have to be an extra layer on Cesium3DTileset that finds which tiles intersect the ray.

6

Sean,

Thank you very much for your reply. I think I am going to try out the ray-triangle approach.

I've done a cursory look for the actual mesh/vertex buffer for a single Cesium3DTile but have not found it. Could you point me in the right direction for accessing this? More generally, outside of the source code itself, I have not seen any resources that dive deep into the inner workings of Cesium. If such a thing exists, I'd love to see it. Thanks again for your help.

Eric

7

You should start by working with Model.js. A b3dm contains a Model instance, so if you can get picking working for models you can get it working for b3dm.

Probably the best course of action is to put some code in createIndexBuffer and createVertexBuffer that saves the indices and position typed arrays. Normally these would get deleted when loadResources is set to undefined, but not if you save them somewhere else. Every 3 values in the indices array will describe a single triangle, where each of these 3 values points to a position in the position array. For example, the position array will be laid out like [x0,y0,z0,x1,y1,z1,x2,y2,z2,x3,y3,z3], so if the first three indices are [0, 2, 3] your triangle be [x0,y0,z0], [x2, y2, z2], [x3, y3, z3]. Then you’ll need to transform each position from its local coordinate system to the world coordinate system. You can use model._modelMatrix to transform each position (one caveat here… check if your model uses the CESIUM_RTC extension, because that will change this step).

That’s the basic approach, let me know if you have questions along the way.

8

Outstanding. I will start on this tomorrow. Thanks for your help.

9

Hi Sean,

Made some headway on this today. I have a couple questions if/when you have time.

1) If every 3 values in the indices array describe a single triangle, shouldn't each value be unique? If the first 3 values were [0, 0, 1], the the triangle would have positions [x0, y0, z0],[ x0, y0, z0], [x1, y1, z1]. In this case 2 of the points that define the triangle would be the same. This doesn't seem right. I ran in to this case today, which is why I ask.

2) Regarding the transformations, the Models that are created from the 3D tileset that I am using all appear to have the identity transformation. I think the CESIUM_RTC is used though. I haven't gotten this far yet, but if CESIUM_RTC is used, in what way will this affect the transform from each position's local coordinate system to world coordinate system?

Have a great weekend.

Thanks,
Eric

10

Hey Eric,

  1. The values in each triangle should be unique… does the model have a single indices array? Also make sure that primitive.mode is 4 (TRIANGLES).

  2. CESIUM_RTC applies an additional translation to the model which needs to get accounted for. I think it should work if you multiply the position by model._computedMatrix and then translate by model._rtcCenter. Are all the node transforms in the glTF itself identity matrices?

11

Hi Sean,

1) My tileset is split across thousands of .b3dm files. I am not sure if this is a valid way to verify how many indices there are in my model, but I spot inspected a few of the .b3dm files and seemed to find there is only one indices array for my model. I inspected the glTF for each of these .b3dm files and found only one mesh with one primitive. The primitive defines POSITION and TEXCOORD_0 attributes and defines a single indices property. Additionally, the mode property is set to 4. This all seems to line up.

I also tried this process out with a different 3D Tiles example I found online, however this tileset did have multiple indices.

For clarity, here is the patch to store the indices and position typed arrays. It is pretty straightforward:

diff --git a/Source/Scene/Model.js b/Source/Scene/Model.js
index a895bb838..6523ce3bd 100644
--- a/Source/Scene/Model.js
+++ b/Source/Scene/Model.js
@@ -1659,29 +1659,31 @@ define([
     CreateVertexBufferJob.prototype.execute = function() {
         createVertexBuffer(this.id, this.model, this.context);
     };

     ///////////////////////////////////////////////////////////////////////////

     function createVertexBuffer(bufferViewId, model, context) {
         var loadResources = model._loadResources;
         var bufferViews = model.gltf.bufferViews;
         var bufferView = bufferViews[bufferViewId];
+        var buffer = loadResources.getBuffer(bufferView);

         var vertexBuffer = Buffer.createVertexBuffer({
             context : context,
-            typedArray : loadResources.getBuffer(bufferView),
+            typedArray : buffer,
             usage : BufferUsage.STATIC_DRAW
         });
         vertexBuffer.vertexArrayDestroyable = false;
         model._rendererResources.buffers[bufferViewId] = vertexBuffer;
         model._geometryByteLength += vertexBuffer.sizeInBytes;
+        model.vertexBuffer = buffer;
     }

     ///////////////////////////////////////////////////////////////////////////

     var CreateIndexBufferJob = function() {
         this.id = undefined;
         this.componentType = undefined;
         this.model = undefined;
         this.context = undefined;
     };
@@ -1696,30 +1698,32 @@ define([
     CreateIndexBufferJob.prototype.execute = function() {
         createIndexBuffer(this.id, this.componentType, this.model, this.context);
     };

     ///////////////////////////////////////////////////////////////////////////

     function createIndexBuffer(bufferViewId, componentType, model, context) {
         var loadResources = model._loadResources;
         var bufferViews = model.gltf.bufferViews;
         var bufferView = bufferViews[bufferViewId];
+        var buffer = loadResources.getBuffer(bufferView);

         var indexBuffer = Buffer.createIndexBuffer({
             context : context,
-            typedArray : loadResources.getBuffer(bufferView),
+            typedArray : buffer,
             usage : BufferUsage.STATIC_DRAW,
             indexDatatype : componentType
         });
         indexBuffer.vertexArrayDestroyable = false;
         model._rendererResources.buffers[bufferViewId] = indexBuffer;
         model._geometryByteLength += indexBuffer.sizeInBytes;
+        model.indexBuffer = buffer;
     }

When I inspect model.indexBuffer after createIndexBuffer has been called, this is the result:

[0, 0, 1, 0, 2, 0, 3, 0, 4, 0, 5, 0, 3, 0, 5, 0, 6, 0, 3, 0, 7, 0, 4, 0, 8, 0, 3, 0, 9, 0, 10, 0, 7, 0, 3, 0, 11, 0, 12, 0, 13, 0, 14, 0, 15, ...]

Similarly, for the example I found online, this is the result:

[37, 0, 38, 0, 39, 0, 37, 0, 250, 1, 251, 1, 37, 0, 39, 0, 102, 3, 37, 0, 109, 4, 250, 1, 37, 0, 102, 3, 109, 4, 20, 2, 37, 0, 58, 4, 38, 0 ...]

As you can see, many of these indices don't seem to define real triangles. I thought that some of these might just be 'noise' or 'outliers' that appear from the program I am using to generate the tiles, but the online example seems to have the same problem. On top of that, I have performed some manual testing by placing visual markers in the scene at the calculated triangle vertex locations (including the transformations) and the results do not look correct (overly large triangles that don't correspond to any geometry on the model). I've also manually tested with Cesium.IntersectionTests by building a ray from the camera to a point on the model where I click and have not been able to trigger a successful intersection. I am a bit stuck on this, and I am wondering if you have any other thoughts?

2) I've implemented the transformation correctly as far as I can tell. Due to the issues above I am not 100% certain.

If I can't get past this soon, I will probably try the ShadowMap approach you mentioned before.

Thank you,
Eric

12

I think perhaps I am not reading the index/vertex buffers correctly. I just realized that both of these vertex/index buffers are Uint8Array TypedArrays. At least with respect to reading from the index buffer, I think you have to take in to account the componentType/bytesPerIndex before reading.

13

Ah, good catch. Getting the correct typed array could look something like this (not tested):

var uint8Array = loadResources.getBuffer(bufferView);
var source = model.gltf.buffers[bufferView.buffer].extras._pipeline.source;
var byteOffset = source.byteOffset + bufferView.byteOffset;
var componentDatatype = ComponentDatatype.UNSIGNED_SHORT; // Hardcoded...
var length = uint8Array.byteLength / ComponentDatatype.getSizeInBytes(componentDatatype);
var typedArray = ComponentDatatype.createArrayBufferView(componentDatatype, source.buffer, byteOffset, length);

``

14

Sean,

I think we're almost there. I think the snippet you provided mostly worked, however I had to store the buffer in a DataView object instead of a specific typed array. This is because I found that one of the source.buffer in my example tileset had a byteOffset that was odd (as in not even), even though the componentType indicated that it was an UNSIGNED_SHORT buffer. I assume that this is ok since it seems to come directly from the model. Here is what I ended up with in createIndexBuffer:

(By the way, how do you get code syntax highlighting/markdown in google groups?)

var loadResources = model._loadResources;
var bufferViews = model.gltf.bufferViews;
var bufferView = bufferViews[bufferViewId];
var uint8Array = loadResources.getBuffer(bufferView);
var source = model.gltf.buffers[bufferView.buffer].extras._pipeline.source;
var byteOffset = source.byteOffset + bufferView.byteOffset;
var length = uint8Array.byteLength / ComponentDatatype.getSizeInBytes(componentType)
var buffer = new DataView(source.buffer, byteOffset, length);

I did almost the exact same thing in createVertexBuffer, however I manually computed the length variable. This is because componentType is not passed in to createVertexBuffer. I've sort of assumed that the vertex buffer that I am using is of type FLOAT. This is just based on inspecting some of the .b3dm files. The componentType for the position buffer is not directly defined, however there is an accessor that references the position buffer whose component type is 5126 (FLOAT). This might be faulty logic, but float makes sense for this type anyways. Since I am assuming it is a FLOAT, I have manually calculated the length of the dataview:

var loadResources = model._loadResources;
var bufferViews = model.gltf.bufferViews;
var bufferView = bufferViews[bufferViewId];
var uint8Array = loadResources.getBuffer(bufferView);
var source = model.gltf.buffers[bufferView.buffer].extras._pipeline.source;
var byteOffset = source.byteOffset + bufferView.byteOffset;
var length = uint8Array.byteLength / 4;
var buffer = new DataView(source.buffer, byteOffset, length);

I also noticed something called a byteStride inside the bufferViews for both the index and vertex buffer. https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#bufferviewbytestride The byteStride for my indexBuffer is 2 which to me indicates a tightly packed Uint16Array, which is exactly how I am accessing the index array. The byteStride for the vertex buffer is 12, which indicates a not tightly packed array of something else. I just noticed the byteStride, so I am not sure how to handle it yet or if it is necessary.

Finally, I wanted to share the code I am using to index into the vertex buffer. I am pretty confident this is correct:

var indices = model.indexBuffer;
var length = model.indexBuffer.byteLength / 2;
var vertices = model.vertexBuffer;
for (var i = 1; i < length; i += 3) {
  var p0 = new Cesium.Cartesian3(
    vertices.getFloat32(12 * indices.getUint16(i*2)),
    vertices.getFloat32(12 * indices.getUint16(i*2) + 4),
    vertices.getFloat32(12 * indices.getUint16(i*2) + 8)
  );
  Cesium.Matrix4.multiplyByPoint(mat, p0, p0);
  if (rtc) {
    Cesium.Cartesian3.add(p0, rtc, p0)
  }

  var p1 = new Cesium.Cartesian3(
    vertices.getFloat32(12 * indices.getUint16((i+1)*2)),
    vertices.getFloat32(12 * indices.getUint16((i+1)*2) + 4),
    vertices.getFloat32(12 * indices.getUint16((i+1)*2) + 8)
  );
  Cesium.Matrix4.multiplyByPoint(mat, p1, p1);
  if (rtc) {
    Cesium.Cartesian3.add(p1, rtc, p1)
  }

  var p2 = new Cesium.Cartesian3(
    vertices.getFloat32(12 * indices.getUint16((i+2)*2)),
    vertices.getFloat32(12 * indices.getUint16((i+2)*2) + 4),
    vertices.getFloat32(12 * indices.getUint16((i+2)*2) + 8)
  );
  Cesium.Matrix4.multiplyByPoint(mat, p2, p2);
  if (rtc) {
    Cesium.Cartesian3.add(p2, rtc, p2)
  }
  model.triangles.push(p0);
  model.triangles.push(p1);
  model.triangles.push(p2);
}

With all of this combined, I am still not quite there. Currently I am running in to a RangeError: Offset is outside the bounds of the DataView after a call to vertices.getFloat32(). I am wondering if this could be because I am not accounting for the byteStride.

Anyways, I've learned a lot so far investigating this, and I appreciate your help.

Thanks,
Eric

15

I usually use the { } button in the text panel to write code snippets.

At a quick glance the code generally looks ok, though I see two problems:

The for loop should start at 0, not 1. This may explain the range error. I think the byteStride is fine.

Any call to getUint16 or getFloat32 needs to pass in true as its second parameter. glTF stores all data as little endian but the DataView functions default to big endian.

So for example:

vertices.getFloat32(12 * indices.getUint16(i*2, true), true),

``

16

Sean,

I don't have any sort of panel to select the { } button or any other editing options.

Thank you for your help with this. I overlooked the endianness, and that was the crux of the problem. I have a working proof of concept.

I'd like to eventually submit a pull request, but before I do that I need to clean it up a bit, and I also wanted to get your opinion on the best way to package this. The modifications to Model.js should clearly stay where they are. The code to actually index into the vertex buffers is currently just in a sample application. It seems like it would be helpful to pull that in somewhere. On top of that, do you have any preference for how to expose this? Perhaps something like Cesium.IntersectionTests.rayModel() would work.

Thanks,
Eric

17

Nice that you got it working. IntersectionTests is probably too low level for this sort of check but I would be fine with exposing it as Model.intersectRay.

I’ll be looking out for your PR. Due to all the other stuff going on we might not have a chance to review right away, but this will be super useful to have.

18

Eric,

Did you ever get to posting a PR on this? I've been hunting for a solution to the same issue, and will go through the code here if you haven't, but it would certainly be a valuable resource if it's up there.

Thanks,

Nathan

19

Hi Sean,

Hope all is well. I just saw this pull request: https://github.com/AnalyticalGraphicsInc/cesium/pull/6934. Looks awesome, and I think Scene.pickPositionFromRay() solves the problem I brought up in this thread. Is that correct?

Thanks!
Eric

20

I think you should be able to do once this PR is merged https://github.com/AnalyticalGraphicsInc/cesium/pull/6934. The code there would be a good reference.

This should be in the next version (1.50)!