Cesium in 2015 - KML

The 2015 kickoff thread has made it pretty clear that KML support is a #1 priority for many of you. I imagine it’s become an even higher priority with the deprecation of the Google Earth API. The good news is it’s a high priority for us as well.

The Cesium team has been working on KML since André Nunes got the ball rolling during Google Summer of Code in June of 2013. A large part of our KML efforts since then had more to do with adding core Cesium features needed to support such a large and diverse specification rather than dealing with the intricacies of the spec itself.

Thankfully, our work will soon start to pay off, as “Phase 1” of our KML support will be released with Cesium 1.7 on March 2nd . I say “Phase 1” because this is just the beginning. I’ve personally been disappointed with every web-mapping application that claims to support KML. Every one I have tried barely scratches the surface of its capabilities, I know the Cesium community can do better and our goal is to be fully compatibility with Google Earth. This includes the official OGC specification as well as support for Google’s own gx extension namespace, not to mention the implementation quirks in Google Earth itself. Since that is obviously a lot of work, we decided it would be better to release features as they are ready (with phase 1 being the core architecture we need going forward).

I know there have been many people building (or trying to build) the Cesium kml branch in the hopes of playing with it before the release. Because of such high demand, I decided to start making beta releases available so it’s easier to get started. You can grab the first one from Google Drive: Cesium-kml-2015-01-08.zip.

Even better, if you want to start playing with KML right now, you can drag and drop your KMZ or KML files to this beta version of Cesium Viewer that I’ve hosted on GitHub (or append the “?source=” parameter to load a url). This version will be updated on a regular basis and it’s configured to proxy everything for maximum compatibility (more on that later).

Loading KML programmatically is just as easy as loading CZML or GeoJSON. I also added a SandCastle example.

viewer.dataSources.add(Cesium.KmlDataSource.fromUrl(‘path/to/kmlOrkmz’));

What (mostly) works

  • Both KML and KMZ formats are supported.
  • All geometry types except for models.
  • Time dynamic geometry and intervals.
  • Ground Overlays
  • Local, remote, and shared styles (not all style options are supported)
  • Basic balloon support (via InfoBox).
  • Rudimentary “one and done” NetworkLink support. (No refreshes, no view parameters, etc…)

What doesn’t work

  • Vector data does not currently conform to terrain and altitude mode is ignored.
  • The visibility tag is ignored, everything is loaded as if were visible.
  • Other than the exception noted above, Network Links are unsupported.
  • Screen Overlays
  • Photo Overlays
  • Super Overlays
  • Hover styles
  • drawOrder
  • Tours and related functionality
  • Your favorite KML feature that I’ve probably forgot to mention, sorry! :slight_smile:

Since KML is a large specification, we would love to hear what parts of it are most important to you in order to help shape the roadmap. We’re also looking for some kickass offline free-to-distribute KML examples that we can include for our Sandcastle demo. If you would like to contribute one, or know of one that could do the job, let us know. (Maybe we can even make a little contest out of it and send out some free t-shirts).

I’ll also reply to this post whenever there are new builds to play with or other significant updates to KML support (even after phase 1). For example, just yesterday I added the rudimentary NetworkLink support as well as polygons with holes. If you don’t subscribe to forum updates, feel free to follow me on Twitter (@matt_amato); I’ll post updates on there as well.

Now about that proxy.

If you are familiar with Cross-Origin Resource Sharing (CORS) and the problems it presents, you can skip to the last paragraph. For everyone else, it’s really important to understand the role it plays in whether a particular KML file will work in Cesium or not; so keep reading!

KML is not very web friendly. It allows for local file system references (which can’t be generally supported by web apps). It includes Collada files which allow for all kinds of stuff that aren’t supported on the web (which is why glTF now exists). Descriptions can have embedded HTML which has to be sanitized to avoid security issues, the list goes on and on. Many of these issues we can work around one way or another, but the worst problem is likely not going away anytime soon.

Many existing KML files make use of NetworkLinks or linked image URLs that reference data on external servers. For example, take the National Weather Services KML generator. In theory, it should be easy to load the URL (which returns a KMZ) into Cesium.

viewer.dataSources.add(Cesium.KmlDataSource.fromUrl(‘http://radar.weather.gov/ridge/warningzipmaker.php’));

Unfortunately if you run this code and you’ll get something similar to this error in the console:

XMLHttpRequest cannot load http://radar.weather.gov/ridge/warningzipmaker.php. No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘http://localhost:8080’ is therefore not allowed access.

For those of you not familiar with ‘Access-Control-Allow-Origin’, it’s a header that allows JavaScript to retrieve files from domains other than the one hosting the site. I won’t go into details, but this roadblock exists to prevent certain types of security attacks. Google Earth doesn’t have this problem because it’s a desktop application and not running in a browser sandbox. If the external server enabled CORS, it would have loaded fine (and securely). Unfortunately most servers never enable this functionality

Now you may think to yourself, that’s okay, I’ll just download the file to my own server or computer and use it from there. Unfortunately, you’ll get the same error as above. Why? It’s because like many KML files, it’s just a stub that include a NetworkLink that points back to (you guessed it) http://radar.weather.gov, which we’ve already learned does not enable CORS.

The only real solution is to use a proxy. What is a proxy? In this use case it’s a simple web-service that runs on your own server that downloads the file for you and passes it through to the app. This works around the CORS issue because the web-service is not running in the local browser sandbox. Cesium ships with a proxy as part of the development server. It’s easy to use and the below will work great in the beta.

var proxy = new Cesium.DefaultProxy(’/proxy/’);

viewer.dataSources.add(Cesium.KmlDataSource.fromUrl(‘http://radar.weather.gov/ridge/warningzipmaker.php’, proxy));

Of course the problem is that if you’re trying to write a general KML viewer, you have no way to tell what servers support CORS and which don’t, which means you have to proxy everything (which can be a load-time performance bottleneck and a strain on the server). What’s worse is that if you’re app is on the internet, you really don’t want to run an open proxy for another set of security reasons that I won’t go into. That being said, the simplistic Cesium Viewer app that I linked to at the beginning of this post does use http://www.corsproxy.com. Here’s how I set it up in Cesium so that I could host my demo on GitHub pages.

var proxy = {

getURL : function(url) {

    var tmp = new Cesium.Uri(url));

    url = 'http://www.corsproxy.com/' + tmp.authority + tmp.path;

   

    if(tmp.query){

        url += '?' + tmp.query;

    }



    return url;

}

};

var viewer = new Cesium.Viewer(‘cesiumContainer’);

viewer.dataSources.add(Cesium.KmlDataSource.fromUrl(‘http://radar.weather.gov/ridge/warningzipmaker.php’, proxy));

Keep in mind that by using a third-party proxy, we are sending all KML server requests through the proxy first which means it can see/manipulate the results on the way to the client. This is obviously a security issue for production applications but is not a problem for the demo. You’ll want to use your own proxy and whitelist for any production Cesium application that needs to load KML files you don’t manage yourself.

Of course the good news is that if all of the KML files and related images/NetworkLinks are on your own servers, there is no reason to use a proxy and everything will work great (as long as you enable CORS).

This email was long overdue and turned out a lot longer than I expected; but hopefully it gets everyone interested in KML on the same page and provides insight into our 2015 plans. I plan on making this the “official” KML thread until we release and I’ll probably turn this post into a full featured blog post at that time. For now, test it out and let us know what you think.

Thanks,

Matt

CORS in network links was something that looked like an issue since I first tried out the early kml builds, so its good to see that it is being acknowledged now.

KML may not be very web friendly, but it has a lot of inertia at this point, and people are fairly familiar with it.

KML has a little bit of everything. I am excited to see a new client that works towards doing the full spec justice.

The following would be great for me:

-handling of visibility
-screen overlays
-handling of altitude mode

It's great to see support for KML on the priority list. Inertia is an issue....it's ubiquitous. KML support will help during a transition to CZML, GeoJSON, etc., and loss of NPAPI support.

Thanks,

TJ

Vector data considering altitude and terrain
Photo Overlay

I think those are important features to implement for full KML support.

It will be good to see gx:LatLOnQuad tag in groundoverlay

Hi,

for us, the most important kml support features we would like to see available :

  • altitude mode
  • visibility tag to show or hide vector data
  • to handle photo overlays

Anyway, great project, keep going !

Elias,

IGN, French mapping agency

Hi,

and tags allow to provide KML data in a tiled way based on user position.

This way vector data (and also 3D models referenced in KML via the tag) may be streamed.

The support of these tags is thus important.

Gilles

Hi,
the most important features I would like to have are:
- Vector data conforming to terrain and altitude mode
- Photo overlays
- Support for balloon styling
- Support for balloon <iframe>,<img>,etc..
Luca

what about feature and the ability to load layers with large amount of polygons.
do you have any estimation about when this feature should be available ?

thanks

Hi, there is an interesting YouTube Video for you

I’ve updated the KML demo at http://mramato.github.io/CesiumViewerKML/ to use the latest Cesium 1.6, (released yesterday in case you missed it).

The new version also disables infobox sanitization so HTML will work inside of balloons.

To answer some recent comments and questions:

Luca: balloon images and related HTML are already supported, Cesium is just “secure by default” so external HTML is sanitized before being displayed. You can configure Cesium to allow it and we’re in the process of making it easier (hopefully for next release). I actually cover this some in the InfoBox/description section of the new tutorial posted yesterday: http://cesiumjs.org/2015/02/02/Visualizing-Spatial-Data/

BalloonStyle will hopefully be March 2nd

Clamping to terrain will be either March 2nd or April 1st.

Region should be implemented as part of full NetworkLink support, and will probably be the first significant feature we add after the initial release. (No time frame though).

Photo overlays are a much more popular request than I thought they would be. We have to do more research before we could figure out how hard they are to implement, but I don’t think it will be near term.

LatLongQuad is also not near term

Hopefully didn’t miss anything and thank you everyone for the continued feedback. With 1.6 released, I’m not dedicating the majority of my time to KML for the March 2nd release, so expect new builds with better support to be posted soon!

Thank you Matt for your answer

Hi Matt,
Michael Ross here at DataBC. Looks like you're making great progress toward full KML support. We use KML as a declarative application definition language. Here is an example:

http://openmaps.gov.bc.ca/kml/BCGov_Physical_Address_Viewer.kml

Here are the KML elements that make it work:

1. <NetworkLink> including unlimited nesting and <viewFormat> support. The network links in the physical address viewer link to our WMS services, our geocoder, and our geographical names service.

2. Support for javascript and HTML links and forms in CDATA element of <BalloonText> and <Description>

3.<Folder>

4.<GroundOverlay>

5. <Placemark> including <LookAt>

6. <Snippet>

7. <Style> and <StyleUrl>. We keep our styles in a separate <Document>

8. <gx:balloonVisibility>

Feel free to use our KML file in your torture testing. Keep up the good work.

In another post there was some discussion about orthometric vs ellipsoid elevation and Geoids.

The KML spec was mentioned with reference to section 6.2 and altitude being defined as the orthometric elevation using the EGM96 Geoid.

KML encoding of every kml:Location and coordinate tuple uses geodetic longitude, geodetic latitude, and altitude as defined in Annex A by the GML Coordinate Reference System (CRS) with identifier LonLat84_5773. Note that altitude is measured from the vertical datum, which is the WGS84 EGM96 Geoid. The altitude measurement (orthometric H) is illustrated in Figure 1.

Is this going to be included in the first kml implementation?

And will there be (or is there currently) a way to hit test the terrain to include the egm96 geoid undulation.

Perhaps the MSL gravity geoid surface could be represented identically to how terrain is represented (LOD tiles each with their own triangle mesh.) Use the same pick function even. Terrain tile L4X5Y4 would coincide with MSL tile L4X5Y4. Cross 2 sides of a triangle to get the direction of gravity in the middle of the triangle. You could then render lines to show the gravity vectors. Maybe have an option to render MSL & terrain & reference ellipsoid surfaces all at the same time, all being translucent, for educational purposes.

I’ve just posted a new KML pre-release: https://drive.google.com/file/d/0BwC7x518HAffdzY5UVI1a1NzeTA/view?usp=sharing

I’ve also updated the live demo at: http://mramato.github.io/CesiumViewerKML/

Unfortunately, the open proxy we were using seems to no longer exist (or is incredibly unreliable) so I had to stop using it. This means that the above demo page can’t load data from non-CORS server (this problem is specific to the above demo).

Here are the most significant changes (in addition to the usually bug fixes and cleanup)

  1. Support Icon heading and hotspot, as well as gx extensions x, y, w, h
  2. Initial support for ExtendedData, only the type is currently supported, the two other types will be added eventually. Extended data is also exposed as entity.kml.extendedData.
  3. Added extrude support to Point and Line geometry
  4. BalloonStyle is now supported, including entity replacement for text templates that reference ExtendedData.
  5. We now pre-process the entity.description to automatically generate links the way Google Earth does. All links open in a new window.
  6. gx:Track and gx:MultiTrack now work much better.
  7. Fixed a problem where entity availability was not being set, causing time-dynamic KMLs to not expose the right interval. Dragging and dropping a time-dynamic KML in Viewer should now work in all cases.

Remember to set InfoBoxViewModel.defaultSanitizer = undefined at the beginning of your app to ensure the InfoBox can actually handle the HTML in KML descriptions. I still hope to improve this before the next release.

And some eye-candy to close things out:

Berwyn, the KML code uses the same exact coordinate conversions as the rest of Cesium, so it will use the WGS84 ellipsoid for conversion from cartographic to earth fixed. All of the data readers share the same underlying Cesium code for any coordinate handling. Support for altitudeMode will be incomplete for the initial release (neither height above MSL or clampedToGround will be fully supported), but I expect them to be added soon after, perhaps in 1.8.

I’m not sure what you mean by “include the egm96 geoid undulation”. The terrain is the definitive authority for elevation and any affect on it by the ellipsoid or MSL should already be baked in. Of course this isn’t my area of expertise so you are better off asking for clarification from Kevin or Alex in the other thread.

Matthew, thanks for the reply.

From what I can tell so far, the terrain provided in cesium is ellipsoid elevations, so shorelines that meet the ocean have elevations that vary by location, In Vancouver where I am it is -19.96 meters, San Francisco -32.25, On the east coast, say Miami, it is -28 meters

http://analyticalgraphicsinc.github.io/cesium-google-earth-examples/examples/groundAlt.html

The egm96 geoid undulation is an estimation of the delta between sea level and the ellipsoid,

The coastline of the southern tip of India is -90 meters below the WGS84 ellipsoid. Since the interior Earth isn’t of uniform density oceans can get pulled into funny shapes. KML absolute altitude mode is distance from egm96 geoid which takes into account these variations. I suppose at first Cesium will only use relativeToGround == relativeToSeaFloor. If absolute is treated as height from ellipsoid it can be off, such as 90 meters off at the tip of India.

To complicate matters it seems that the distance from egm96 geoid is along the normal of the geoid, not along the normal of the ellipsoid at that lon/lat. This could potentially allow 2 different lat/lon/alt tuples to describe the same position in space if the geoid curved in a concave manner. Although the ellipsoid and geoid normals shouldn’t be that far off from eachother.

I know it’s only been a few days, but I’ve made a ton of progress on KML support. Here’s another pre-release build: https://drive.google.com/file/d/0BwC7x518HAffb0xmc1V0VGZnenM/view?usp=sharing

As always, I appreciate any feedback you guys have.

Highlights:

  1. loadKmz is gone, you can now call load for both parsed documents or blobs.

  2. Descriptions that reference images and other data inside of a KMZ file are now supported.

  3. Descriptions now have plain text turned into links like Google Earth does

  4. gx:Track and gx:MultiTrack now match Google Earth.

  5. gx:LatLonQuad now has basic support, (texture is not reprojected and won’t be for a while).

  6. Fixed an issue where the load promise resolve before loading was complete.

  7. Fixed an issue with resolving network links via a proxy.

  8. Fixed issues with multi-geometry styling

  9. Fixed extruded linestring outline and fill settings.

  10. Fixed PolyStyle defaults to match GE

  11. Start altitudeMode support for Point geometry (currently clampToGround clamps to ellipsoid)

  12. Support for atom:author, atom:link, address, phoneNumber, and Snippet.

Just a reminder that you can follow overall KML progress at https://github.com/AnalyticalGraphicsInc/cesium/issues/873

I’ve also opened a PR for 1.7 into Cesium master: https://github.com/AnalyticalGraphicsInc/cesium/pull/2503

Thanks,

Matt