How to efficiently extract 3D coordinates for all pixels?

CesiumJS
1

Hi everyone, I have one question about how to extract 3D coordinates (latitude, longitude and height) for all pixels in the camera view in the fastest way.

I understand I posted a similar question before but this one is about how to improve efficiency (last one on how to obtain 3D coordinates). The problem is that a large number of images + coordinates per pixel are needed (e.g. 10,000 images at 600*400 + all coordinates). In the hardware accessible to me, the speed was like 20s per image + coor. while it took only 3-5s for one RGB image without the coordinates. Since the processing speed is really a limit, I was wondering if there is any way to extract coordinates more efficiently.

Our code is like this:
for(var ii=0;ii<viewer.canvas.width;ii++){
for(var jj=0;jj<viewer.canvas.height;jj++){
var cur_pos = new Cesium.Cartesian2(ii, jj);
// use scene.pickPosition
var ppCartesian = viewer.scene.pickPosition(cur_pos);
if (Cesium.defined(ppCartesian)){
var ppCartographic = Cesium.Cartographic.fromCartesian(ppCartesian);
var s = Cesium.Math.toDegrees(ppCartographic.latitude).toFixed(7);
stringCart += “[(” + s.toString() + “,”;
s = Cesium.Math.toDegrees(ppCartographic.longitude).toFixed(7);
stringCart += s.toString() + “,”;
s = ppCartographic.height.toFixed(0);
stringCart += s.toString() + “), “;
stringCart += “(”+ii.toString()+”,”+jj.toString()+“)], "
}
else{
stringCart += “[(-1,-1,-1), “;
stringCart += “(”+ii.toString()+”,”+jj.toString()+”)], "
}

Cesium 1.63 and firefox were used. Looking forward to your opinions. :slight_smile:

2

Can you re-post what it was you needed all the coordinates for? Is the goal to do this for every frame in real time?

It sounds like it may be significantly more efficient to just output the contents of the depth buffer, which will color each pixel by the Z coordinate, if you can work with that output.

3

Thanks for your message. Yes the depth buffer is very helpful as well! I saw that many functions in Scene class use the depth buffer, and I was wondering if there is any function to get depth buffer directly?

The intended goal is to have the 3D coordinates per image so that we could feed them into the deep neural network to do camera pose localization, and the coordinates provide distinctive and good features. Being able to obtain these information efficiently play an important role in improving the performance of our algorithm (e.g. better accuracy in a larger area). It doesn’t have to be real-time but needs to be reasonaly fast (not like 20s per image right now).

4

I think you may need to do some digging in the source code to get the depth buffer. There’s some discussion on this here: Postprocessing with normal and depth information and a more recent post here: How to Get DepthTexture of Current Scene?.

If you do figure out a straightforward way to extract it, it’d be great to post your solution here, or contribute a helper function of some kind to the library for this use case!