Cesium ION 1.9.0 Asset Server crashes with custom client code

Cesium ion
1

We have crashes on latest Cesium ION 1.9.0 Asset Server:

     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a                          │
│ net.(*netFD).Read(0xc0004cfd80, {0xc000247ea1?, 0x0?, 0x0?})                                                                                   │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/fd_posix.go:55 +0x25                                     │
│ net.(*conn).Read(0xc00028c148, {0xc000247ea1?, 0x0?, 0x0?})                                                                                    │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/net.go:194 +0x45                                         │
│ net/http.(*connReader).backgroundRead(0xc000247e90)                                                                                            │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/http/server.go:690 +0x37                                 │
│ created by net/http.(*connReader).startBackgroundRead in goroutine 321                                                                         │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/http/server.go:686 +0xb6                                 │
│                                                                                                                                                │
│ goroutine 339 [select]:                                                                                                                        │
│ database/sql.(*DB).connectionOpener(0xc000225110, {0x9de960, 0xc0001f1130})                                                                    │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/database/sql/sql.go:1261 +0x87                               │
│ created by database/sql.OpenDB in goroutine 321                                                                                                │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/database/sql/sql.go:841 +0x130                               │
│                                                                                                                                                │
│ goroutine 254 [IO wait]:                                                                                                                       │
│ internal/poll.runtime_pollWait(0x7f9a64181490, 0x72)                                                                                           │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/runtime/netpoll.go:351 +0x85                                 │
│ internal/poll.(*pollDesc).wait(0xc0001ffa80?, 0xc000219bd1?, 0x0)                                                                              │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27                    │
│ internal/poll.(*pollDesc).waitRead(...)                                                                                                        │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/internal/poll/fd_poll_runtime.go:89                          │
│ internal/poll.(*FD).Read(0xc0001ffa80, {0xc000219bd1, 0x1, 0x1})                                                                               │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a                          │
│ net.(*netFD).Read(0xc0001ffa80, {0xc000219bd1?, 0xc00000ea58?, 0x1?})                                                                          │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/fd_posix.go:55 +0x25                                     │
│ net.(*conn).Read(0xc000196250, {0xc000219bd1?, 0x0?, 0xc00000ea60?})                                                                           │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/net.go:194 +0x45                                         │
│ net/http.(*connReader).backgroundRead(0xc000219bc0)                                                                                            │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/http/server.go:690 +0x37                                 │
│ created by net/http.(*connReader).startBackgroundRead in goroutine 199                                                                         │
│     /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.13.linux-amd64/src/net/http/server.go:686 +0xb6

another:

github.com/CesiumGS/asset-server/lib.(*TerrainReader).checkParentTile(0x8d70e0?, {0x8b1260?, 0x9e9810?}, 0x14?, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:266 +0x2c5
github.com/CesiumGS/asset-server/lib.(*TerrainReader).hasTile(0xc000266af0, 0xf, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:240 +0x126
github.com/CesiumGS/asset-server/lib.(*TerrainReader).checkParentTile(0x8d70e0?, {0x8b1260?, 0x9e9810?}, 0x14?, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:271 +0x30f
github.com/CesiumGS/asset-server/lib.(*TerrainReader).hasTile(0xc000266af0, 0xf, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:240 +0x126
github.com/CesiumGS/asset-server/lib.(*TerrainReader).checkParentTile(0x8d70e0?, {0x8b1260?, 0x9e9810?}, 0x14?, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:271 +0x30f
github.com/CesiumGS/asset-server/lib.(*TerrainReader).hasTile(0xc000266af0, 0xf, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:240 +0x126
github.com/CesiumGS/asset-server/lib.(*TerrainReader).GetTile(0xc000266af0, 0xf, 0x85b2, 0x61b7, {0xc000080500, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainReader.go:408 +0xbb
github.com/CesiumGS/asset-server/lib.(*TerrainTilesetDatabase).GetTile(...)
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/lib/terrainTilesetDatabase.go:28
github.com/CesiumGS/asset-server/handlers/terrain.GetTerrainTileImplementation({0x7f776a461ad8, 0xc00003e480}, 0xc00041cf40, 0xc000124620)
    /home/runner/go/pkg/mod/github.com/!cesium!g!s/asset-server@v1.7.1/handlers/terrain/getTile.go:48 +0x252
github.com/CesiumGS/cesium-ion-self-hosted/asset-server/lib.routeTerrainRequest({0x7f776a461ad8, 0xc00003e480}, 0xc00041d100, 0xc000124620, {0xc0002d80e7, 0x16})
    /home/runner/work/cesium-ion-self-hosted/cesium-ion-self-hosted/packages/ion-asset-server-go/lib/routeFromUri.go:149 +0x608
github.com/CesiumGS/cesium-ion-self-hosted/asset-server/lib.RouteFromUri({0x7f776a461ad8, 0xc00003e480}, 0xc0006a6b40, 0xc000124620)
    /home/runner/work/cesium-ion-self-hosted/cesium-ion-self-hosted/packages/ion-asset-server-go/lib/routeFromUri.go:79 +0x5d2
main.main.func7({0x7f776a461ad8?, 0xc00003e480?}, 0xc0000acab0?)
3

Thank you for reporting this, we’re looking into it.

-Ben

4

@remoe Can you please share any steps that lead up to this crash? Did this occur when attempting to stream data or without any streaming? From our understanding this arose once you have upgraded from 1.8 to 1.9, any other changes on your end? Please correct our understanding.

Thank you.

5

This issue occurred in versions 1.8 and 1.9 when the app streams data from a self-hosted ION server. Below is an example URL that triggers a 502 error:

https://our-ion-server.local/assets/asset_depot/8/imported/cesium_world_terrain_v1.2.terraindb/15/34417/24929.terrain?extensions=octvertexnormals-metadata&v=1.48133.0
Request Method
GET
Status Code
502 Bad Gateway

Add of “–log-level debug” does not change the log-level.

6

I was able to reproduce an asset-server panic with a faulty custom Cesium composite terrain provider. The client incorrectly advertised terrain tile availability and then requested missing tiles. This appears to drive TerrainReader.checkParentTile/hasTile into a server-side crash path. The client is at fault, but the server should probably fail gracefully instead of panicking ?