Distance to bounding sphere function looks wrong

[I was about to file a bug report on github, but guidelines said post here instead. Hi everyone! First post, hope it comes out right, I didn’t see a “preview” button]

I noticed this when investigating Issue #9534 .

BoundingSphere.distanceSquaredTo() currently looks like this:

BoundingSphere.distanceSquaredTo = function (sphere, cartesian) {
  //>>includeStart('debug', pragmas.debug);
  Check.typeOf.object("sphere", sphere);
  Check.typeOf.object("cartesian", cartesian);
  //>>includeEnd('debug');
  
  var diff = Cartesian3.subtract(
    sphere.center,
    cartesian,
    distanceSquaredToScratch
  );
  return Cartesian3.magnitudeSquared(diff) - sphere.radius * sphere.radius;
};

Shouldn’t the last line be (Cartesian3.magnitude(diff) - sphere.radius)**2, instead?

Plotting correct vs computed distances from a unit-radius sphere:

gnuplot
  set samples 10000
  set parametric
  r = 1  # can try other values, but the graph looks the same
  set xlabel "correct distance"
  set ylabel "computed distance"
  plot [0:3*r] [0:2*r] [0:2*r] t-r,t-r, t-r,sqrt(t**2-r**2)

How I expect this manifests when viewing a 3dtiles tileset (though I haven’t tried it):

  • When far from the planet, the formula results in a computed SSE that’s approximately correct.
  • But when zooming in closer to the planet, the computed SSE becomes progressively smaller
    than the real SSE, infinitely-times smaller in the limit.
    That means the user will experience the detail becoming progressively worse/blurrier,
    and if ey have a knob controling maxSSE, ey will have to keep dialing it down (finer)
    more and more as ey zoom in, to compensate.

If this is correct, it seems “sphere” bounding volumes cannot reliably be used in a 3dtiles database, until this is fixed.
(And “region” boundingVolumes don’t work either until Issue #9534
is fixed; so the only trustworthy boundingVolume type at the moment seems to be “box”.)