GE KML Migration Challenges

CesiumJS
1

Hi,

First let me qualify up front I’m very new to GIS space but learning quickly! We have a client that built a very large KML (120MB) file for Google Earth to serve over the Web and we were tasked to find a way to improve performance. Obviously downloading and the loading a file that large was not going to be a happy experience for any user. So, my first attempt was to find a way to break the file up and serve it more efficiently. But the deprecation of the GE plugin pretty much forced us to look at other 3D solutions. Thus Cesium.

I’ve created a stack of Cesium/Geoserver/PostGIS/PostgreSQL as I think this will provide the most optimum way to manage and serve this fairly complex visualization, especially as they plan to extend it going forward. I used Google Earth desktop to break the KML out to smaller files. I have some challenges with breaking down some of the layers there but I’ll leave that out of discussion here unless you can suggest a better way to break down the KML before importing into PostgreSQL. I am using ogr2ogr to import the file into to PosgtreSQL directly with the following command "ogr2ogr -f “PostgreSQL”. Some web resources suggested to convert to a shapefile first but I’m not sure that would create a different result? Seems to go pretty well so far. The issues I’m having comes to serving up the layer via WFS in Geoserver to Cesium using geoJSON and rendering using GeoJsonDataSource(). Below are 3 challenges I’m having and would like some suggestions or let me know if Cesium currently doesn’t support it. BTW I’ve tried using the KML branch to serve the KML directly but wasn’t having any luck. Regardless I think serving on the above stack is a better overall solution and if I can import into the stack then I thought geoJSON would be a better option to serve the layer than KML.

Challenge #1.

As a first attempt to validate the stack I imported the sample geoJSON file in the SampleData folder from the Cesium download. When serving that as geoJSON from Geoserver it changed the syntax of the geoJSON by replacing dashes “-” with an underscore “_”. Example property attribute “marker-symbol” became “marker_symbol”. Cesium did not understand this and the layer did not render. This may be a Geoserver issue but I’m at a loss how to fix it or if there is some variation in the spec that isn’t supported.

Challenge #2

When importing the legacy KML into PostGIS I used EPSG 4326 as the coordinate system. Subsequently, Geoserver adds a CRS object to the end of the geoJSON data that looks like this

“crs”: {

“type”: “name”,

“properties”: {

“name”: “urn:ogc:def:crs:EPSG::4326”

}

}

However, this won’t render in Cesium. But if I remove the CRS object from the geoJSON (I’m serving from a file as a testing ground in the Sandcastle app), it renders. But I’m not sure correctly as when I zoom in it seems the lines are a little off.

Any clue? Is the structure invalid? Also, if I replace the URN with this one “urn:ogc:def:crs:OGC:1.3:CRS84”, it renders fine with the same “offness” as removing it entirely.

Challenge #3

All the imported KML that I’ve successfully imported renders in Cesium as the color Yellow. It does not seem to retain the colors specified in the KML. I haven’t spent much time trying to troubleshoot this yet but was wondering if the conversion process just lost this data or if I’m doing something wrong. Again, this might not be a Cesium issue but maybe someone can offer some clue. Or do I need to modify the imported data after the fact? If so, should I just do that in the Cesium API? Or should I do this in Geoserver? If the latter, what’s the best way to do this? Do I need a mapping software like OpenLayers to do this?

Sorry for the novice questions, but this is a lot to learn! If this is not the appropriate place to ask some of these questions, please direct me to a better source. Is something like gis.stackexchange.com a good place?

BTW here is my Cesium code in Sandcastle if that helps.

var viewer = new Cesium.Viewer(‘cesiumContainer’);

Cesium.viewerEntityMixin(viewer);

var dataSource = new Cesium.GeoJsonDataSource();

var url=’…/…/SampleData/simplestyles.geojson’

dataSource.loadUrl(url);

viewer.dataSources.add(dataSource);

NOTE: I’m just copy and pasting the Geoserver WFS served data into the simplestyles.geojson file as a way to test temporarily. Easier than dealing with the proxy server issues for now.

Thanks

Joe

2

Hi

I also had some trouble importing complex KML files (involving .KMZ, network links, styles, etc..) to PostGIS/GeoJSON* with Ogr2Ogr. Ogr2Ogr actually has 2 KML implementations ("KML" is usually available by default but "LibKML" is more complete).

What seemed to work better was using a KML/KMZ reader like GisCore https://github.com/OpenSextant/giscore, for Java, as a preprocessing stage. It can unzip .kmz and recurse NetworkLinks transparently.

We found GeoServer limiting and unnecessarily tedious to configure. Constellation https://github.com/Geomatys/constellation is a little prettier and sensible - but neither quite as flexible as working directly with a GIS index directly through a supporting API. ex: PostGIS, and even Lucene via Solr or ElasticSearch.

3

Thank you. I’ll have to check Constellation out. I’ll also give the GisCore reader a shot.

Still need a good answer for my CRS challenge as that is the biggest potential showstopper for me.

Thanks!

Joe

4

KML is a large and complicated (and sometimes ill defined) spec and is also not very browser-friendly. Any kind of preprocessing to another format will significantly degrade the quality of the data and visualization in one form or another. For example, many NetworkLinks are “live” and return different data based on time of day or current view, so a “one and done” conversion will never give you what you want. Going to GeoJSON makes things even worse because it handles only a trivial amount of the overall KML feature set (and has no de facto styling standard). That being said; conversion is a viable approach if you are only using a subset of KML features and have total control over your data set (i.e. you aren’t importing KML files created by others). I’ll also add that if you have the source data and our generating KML from it yourself, it may be easier to generate CZML instead, or write a custom Cesium DataSource that goes directly to web-services rather than using data files stored on the server. It all depends on your use case.

Cesium’s goal is to have full KML compatibility, both with the official OGC KML standard as well as Google’s own gx extensions. The kml branch in Cesium currently handles a lot of the core geometry features (which is great for a large subset of KML files), but not much else. I’ll be switching my focus exclusively to finishing up KML phase 1 soon, so expect some major mailing list announcements after the New Year. I’ll also start putting up beta-builds for people to download and play with at that time. You can see our roadmap here: https://github.com/AnalyticalGraphicsInc/cesium/issues/873 (ignore the checkboxes for now).

To respond to your main bullet points

  1. If you can share the GeoJSON file that isn’t rendering in Cesium, I can take a quick look and tell you if it’s valid or not. If it’s valid, then it’s a bug in Cesium and I’ll fix it; if it’s not, then it’s a bug somewhere else in the process. Going from marker-symbol to marker_symbol is definitely a bug in GeoServer but does not invalidate the file itself, it will just cause the styling to go away (because marker_symbol is not part of the simplestyle standard).

  2. The only two CRS’s that are configured out of the box are urn:ogc:def:crs:OGC:1.3:CRS84 and EPSG:4326 (as you can see at the top of the source file here: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/DataSources/GeoJsonDataSource.js) You can easily register your own conversions though

Cesium.GeoJsonDataSource.crsNames[‘urn:ogc:def:crs:EPSG::4326’] = function (coordinates) {

return Cesium.Cartesian3.fromDegrees(coordinates[0], coordinates[1], coordinates[2]);

};

if urn:ogc:def:crs:EPSG::4326 is a common enough entry (and is identical to the current 2 standards as it apears to be), we can add it to the defaults as well. As far as the wavyness of lines, are you using terrain? Geometry in Cesium does not currently support terrain draping (but as I said above, this is planned). So the lines are actually under the terrain and will appear to move oddly relative to the camera. If you shut off terrain, is it correct? If this isn’t the problem; then can you post the file for me to take a look at?

  1. I already answered this one accidentally. GeoJSON has not standard styling, the closed you get is the simplestyle standard (which Cesium does support). It sounds like GeoServer isn’t supporting that though. There’s no way to define certain geometry (like extruded polygons) in GeoJSON, so it’s simply not a viable alternative to KML in general.

I hope that helps; let me know if you have any additional questions. In short, my official answer is if you have or want to use KML, native KML support in Cesium is your best bet and please report any problems or issues you have when trying to use the kml branch.

Thanks,

Matt

5

Matt,

Thanks for the detailed reply! I’ve actually learned quite a bit more since I posted! Many late hours :slight_smile:

The KML we have, while large, is static and self contained without network links. I’ve been able to successfully migrate a couple layers so far. Unfortunately, we don’t have the source. I’m hoping that there isn’t a lot of loss as most of it is just lines, polygons and points. I’m hoping what doesn’t come over I can make up for in code as long as the geometry makes it. I’ve already figured out how to create styles in Geoserver and associate with the layer so I can manage it from a central location than in code. I’ve already written a prototype Cesium app that retrieves the files via web services and parses them. That’s how I have managed to get around the geoJSON issue. I used the loadJson(url) method and then removed the CRS object and then added styles and then added it to the viewer datsource. Seems to work well so far.

  1. I attached the output of the simplestyles.geojson conversion. I don’t know if it is Geoserver or the ogr2ogr conversion that does this. But the file is attached.

  2. I managed to get around this by loading and then removing the CRS property. I have no idea why Geoserver uses a different notation for EPSG:4326 but what I showed you is how it is output. You can see a full example in the simpleStyles conversion file I attached as well. I made a mistake regarding wavy lines. I just happened to pick a feature that was drawn incorrectly in the first place! After comparing it back to how it renders in Google Earth I see that they look the same, so I think we are good here. If I remove the CRS from the geoJSON and it uses the default, is that CRS84? Will it look any different than EPSG:4326? So far I haven’t noticed any after accounting for poor drawing :slight_smile: I’m not using terrain mapping so I think we are good here. I’ll know more as I convert the other layers.

  3. Yeah I figured that out after reading some more. I thought it was odd that the spec didn’t include styling but I guess it must have originated before mapping programs. It does make sense to include styling as an extension though but then Geoserver and such would need to support that. As I said I came up with a solution that uses SLDs from GeoServer that I can associate with a layer. I parse it in Cesium and add it to the the datasource during the load process.

I would really like to keep the solution as complete management stack, so keeping it in KML isn’t really necessary since I have to get it into PostGIS and Geoserver anyway. I’ll post back if I have further issues with this process. It may take me awhile! :slight_smile: But I’ll say this is an interesting space. Glad I’m developing these skills :slight_smile:

My biggest issue right now is trying to figure out how to block the Cesium.loadXML call in a while loop so I can process the results. It’s driving me crazy. I don’t know if this is a lack of Javascript understanding or the API but I can’t get it to work. Essentially I have loaded a list of layers using loadXML from a web service and then iterating across those layers to get the layer data and styles in a while loop by making another loadXML call to a web service. But the while loop just moves on past while the call is being executed and the normal ways to block just don’t seem to work. If you have any examples of how this might work, I’d appreciate that!

Thanks!

Joe

simplestyles3.geojson (40.1 KB)

6

Since no one’s responded yet, a quick bit of advice: what I assume you’re seeing is the difference between asynchronous and synchronous code. Your call to Cesium.loadXML() starts an async call that needs a callback function to handle the response. Meanwhile, the current chunk of code that the JS interpreter is executing moves on to the while loop, which presumably is looking at an empty array of some sort since there’s no way for the async request to have returned with data while the current line of execution is going on.

Suggest doing some reading up on JS sync/async behavior. Also, note that Cesium uses a concept called “promises” in several places, including as a return result from Cesium.loadXML(). Here’s a few good primers on some of these topics:

http://jessewarden.com/2014/10/message-systems-in-programming-callbacks-events-pub-sub-promises-and-streams.html
http://tech.pro/blog/1402/five-patterns-to-help-you-tame-asynchronous-javascript

Hopefully that gives you enough info to get you going in the right direction.

Mark Erikson

7

Thanks Mark

My problem was that the loadXML calls do not take parameters that can be passed to the callback. I figured out how to pass the parameters I needed in the retrieved document. This allowed me to chain the calls appropriately which is a better solution anyway as it will allow each layer to load independently.

Thanks!

Joe