Failing to load World Terrain from Cesium Ion, because of `traceparent` request header, automatically set by OpenTelemetry Web Auto Instrumentation

Cesium ion
1

Hi all,

Our requests to load the Cesium World Terrain are rejected by the access control checks of the browser, because of the presence of a traceparent header in the XMLHttpRequest emitted by our OpenTelemetry Auto-instrumented web app.

For those GET requests, the browser console tells us the following, :

[Error] Request header field traceparent is not allowed by Access-Control-Allow-Headers.
[Error] XMLHttpRequest cannot load https://api.cesium.com/v1/assets/1/endpoint?access_token=‹redacted› due to access control checks.
[Error] Failed to load resource: Request header field traceparent is not allowed by Access-Control-Allow-Headers.

Question

What could we do to have Cesium Ion serve the World Terrain to our OpenTelemetry-enabled web app?

Specifically, would it be possible to instruct Cesium Ion to accept the traceparent header? for instance, configure an Asset Token which would deliver traceparent in the Access-Control-Allow-Headers response headers?

Has anyone encountered the same issue? Would there be another way?

Cause

Rejection is currently caused, as I understand, because Cesium Ion does not set the traceparent header set in its Access-Control-Allow-Headers response header.

Further, we are using OpenTelemetry Web Automatic Instrumentation [1], which patches fetch and XMLHttpRequest, to automatically inject a traceparent header [2] in all requests made by a web app.

Motivation – So why use traceparent headers?

We want to use such automatic instrumentation, because we are using (great) test tools such as Tracetest [3], to write integration tests of our web app, which we can run against the traces emitted by our app at runtime.

Traces are kind of log messages – so we run our tests against these structured log messages.

Testing against traces, also coined «Observability-Driven-Development (ODD)» [4], is a powerful approach to development and debugging our 3D web apps in production, where the runtime environment has big impact and is out of our control:

when a customer reports «something is not working», we can ask them to activate tracing — they enter a &trace query parameter to the web app URL, and communicate us the generated traceparent id — and we can run the tests remotely, against the collected traces, to discover what actually is failing in their runtime environment.

[1] Automatic Instrumentation | OpenTelemetry
[2] https://github.com/w3c/trace-context/blob/main/spec/20-http_request_header_format.md
[3] https://tracetest.io
[4] https://www.youtube.com/watch?v=0JQrQooLSc8

2

Hi,

Thank you for writing up the issue you are facing in such detail. We want to support the use cases of our users as much as we can. Hence, I have opened an internal ticket for the team to take a look at adding this header and evaluating if we will eventually support it. However, since this is changing our allowed headers, the team may take some time to evaluate it.

I have linked this forum thread to the internal issue, so that you will be updated as soon as the team has something to report.

Thanks,
Ankit

1 Like
3

Hi Ankit,

Many thanks for looking into this and opening the ticket internally.

We have a repository on GitHub, with a project that showcases the behaviour I described. If anyone from your team would like to see it, please send me their GitHub account names and I’ll grant them access to the repo.

This showcase repository also contains the Docker Compose stack with the few containers need to get Tracetest running, with an OpenTelemetry and a Jaeger UI containers, to collect and inspect traces.

We could also schedule a visioconference and I can quickly show and walk you trough our usage of telemetry with Tracetest. We’re in CET timezone (Paris/Frankfurt/Zürich).

Olivier

4

One very valid reason for which any platform, hence your Cesium Ion platform, might want to reject incoming requests with a traceparent set, is the risk of information exposure [1], that is, accidentally leaking your own telemetry traces – would you use them.

To mitigate such risk, one solution is to replace the incoming traceparent header with a new trace id, you’d regenerate for your own downstream requests, to not propagate the incoming trace id.

In any case, as I understand, this information exposure risk however mainly concerns corporate environments. In our case, would the Cesium Ion platform also collect telemetry traces, you’d send the telemetry traces to your own collector, on your infrastructure – and there would be no risk that the collector of our web app receives any of your traces.

[1] Trace Context › 7. Security Considerations › §7.1 Information Exposure

5

An update: in my experiments, these Cesium API endpoints caused the browser to reject the traceparent headers, for requests we’re trying to (auto) instrument and trace; those are:

Sample failing URLs:

6

And here’s a workaround, would anyone run into this problem – but it has the effect of disabling tracing of all requests to the Cesium API [1]

import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web';

class WebTracer {
  […[
  _traceparent;
  _tracerProvider;
  _instrumentations;

  […]
  constructor(traceActivated = true) {
    this._tracerProvider = WebTracer.createWebTracerProvider(traceActivated);
    this._instrumentations = WebTracer.registerWebInstrumentations(this._tracerProvider);
    this._traceparent = WebTracer.extractParentTraceId();  // TODO: use TRACEPARENT instead?
  }

  […]
  static registerWebInstrumentations(provider) {
    const webAutoInstrumentations = getWebAutoInstrumentations({
      '@opentelemetry/instrumentation-xml-http-request': {
        propagateTraceHeaderCorsUrls: [ /.+/g, ],
        ignoreUrls: [
          /api.cesium.com\/v1\/assets\/\d\/endpoint/,
          /assets.ion.cesium.com\/us-east-1\/asset_depot\/\d/
        ]
      },
      '@opentelemetry/instrumentation-fetch': {
        propagateTraceHeaderCorsUrls: [ /.+/g, ],
        ignoreUrls: [
          /api.cesium.com\/v1\/assets\/\d\/endpoint/,
          /assets.ion.cesium.com\/us-east-1\/asset_depot\/\d/
        ]
      },
    });

    return registerInstrumentations({
      tracerProvider: provider,
      instrumentations: [ webAutoInstrumentations ],
    });
  }

The trick is to define the ignoreUrls array in the configuration of the @opentelemetry/instrumentation-xml-http-request and @opentelemetry/instrumentation-fetch (the later is not needed, Cesium actually only performs XMLHttpRequests):

        ignoreUrls: [
          /api.cesium.com\/v1\/assets\/\d\/endpoint/,
          /assets.ion.cesium.com\/us-east-1\/asset_depot\/\d/
        ]

Hope this helps!

[1] Therefore we can’t write tests checking that our app triggers Cesium.js to request the CesiumWorldTerrain on a known URL for instance, without explicitly adding code to instrument and trace around those requests.

One interesting benefit of auto instrumentation of all fetches and XMLHttpRequests is that it allows QA to write such tests independently, after the code was released, to investigate issues, knowing that every single HTTP/HTTPS request is being instrumented, from any API.

Applying the above mentioned ignoreUrls workaround defeats this ability of QA to write tests against production traces, independently, after the code was released. (Sorry for the repetition, I am trying to establish why we like auto-instrumentation and why we still have a problem here, even though there’s a work-around.)