Connect cesium ion to own S3 bucket with REST API

Hi,
I’m trying to use the REST API to create an asset to cesium ion, then upload the file to my bucket, then tell cesium ion where the file is so it can do the tilling.
If I understand correctly, I have to tell on the first step, when I create the asset in cesium ion, that the file is hosted on my bucket, using the “from” object.
My problem is, I can only create the asset if I don’t have the from part.
This is not working (Request failed with status code 400):

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: '3DTILES',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      },
      from: {
        type:'S3',
        bucket: 'mybucket',
        credentials :{
          accessKey:'key',
          secretAccessKey:'secretkey'
        }
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

This is working :

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: '3DTILES',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

but then when I want to tell ion the file has been uploaded to S3, I get an error, because he doesn’t understand the file is in mybucket, not cesium bucket.
What am I doing wrong with from ?
Thanks !

I don’t see the prefixes listed in your from object. See the example on the right hand side of Ion REST API documentation – Cesium.

Hi @Shehzan_Mohammed,
Ok, so I have tried like this :

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: 'POINT_CLOUD',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      },
      from: {
        type:'S3',
        bucket: 'my',
        prefixes: ['myprefix'],
        credentials :{
          accessKey:'mykey',
          secretAccessKey:'mysecretkey'
        }
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

Now I get a 409 error.

Sorry, my bad. it’s working with prefixes. Thanks !

Hi @Shehzan_Mohammed,

Ok so now I create my assets, upload it to s3, but when I want to tell ion that it’s done, I get a 409.
I’m pretty close, I can feel it, but not there yet :smiley:

This is what is answered when I create the asset :

{
    "data": {
        "assetMetadata": {
            "id": 1198055,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-11T09:43:23.636Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-length": "184",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer mytoken"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"mybucket\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"secret\",\"secretAccessKey\":\"mysecret\"}}}"
    },
    "request": {}
}

So now I try to tell ion the upload is finished like this :

var body = onComplete.config.data;
    axios.post(onComplete.config.url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });

But then I get a 409.
I guess there is something wrong with the body var, but I can’t figure out how it should be.
Thanks for the help !

Have you taken a look at cesium-ion-rest-api-examples/index.js at main · CesiumGS/cesium-ion-rest-api-examples · GitHub?

Yes, but when I look at the answer from the call that create the assets in ion, when I use from, I don’t have the onComplete object.
Here is what I get :

{
    "data": {
        "assetMetadata": {
            "id": 1198632,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-11T14:40:31.257Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-length": "184",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer mybearer"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"lidarvisor\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"mykey\",\"secretAccessKey\":\"mysecretacceskey\"}}}"
    },
    "request": {}
}

When I don’t use “from” in the first call, I get a response with an “onComplete” object like in the example you give, with an url.
Here it’s missing.

A 409 requests indicates some type of conflict. From your original comment in Connect cesium ion to own S3 bucket with REST API - #3 by romza, here are the issues I see:

  • type should be 3DTILES. The type indicates the type of output, not input. Use the options.sourceType as POINT_CLOUD instead.
  • Regarding options.position - if your point cloud already has a CRS embedded, then this is not needed.

These are the ones visible to me. You may need to double check your AWS keys, but I can’t confirm that from my end. I would recommend using the Web browser to replicate this upload using the same options (keys, buckets, prefix etc), ensure that upload is successful, and then using the CLI.

Hi @Shehzan_Mohammed ,
Thanks for the help, you were right about the type, it should be 3DTILES.
But my problem is not to upload the file to my bucket, this is working, so there is no issue with my AWS keys.
It’s when the upload is done, I’m missing the url to tell ion it can start tilling.
I tried :

    var body = onComplete.config.data;
     axios.post(onComplete.config.url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });

Since I’m missing the onComplete url, I tried with the config.url (https://api.cesium.com/v1/assets)
But I get a

POST https://api.cesium.com/v1/assets 409

So I tried to recreate the onComplete url like this :

var id = onComplete.data.assetMetadata.id;
var url ='https://api.cesium.com/v1/assets/'+id+'/uploadComplete';
var body = onComplete.config.data;
axios.post(url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });
   

but then I get a

POST https://api.cesium.com/v1/assets/1207043/uploadComplete 400

My problem is that when I post to https://api.cesium.com/v1/assets, if I don’t use from, I get in return this :

{
    "data": {
        "assetMetadata": {
            "id": 1207176,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-14T10:19:01.128Z",
            "status": "AWAITING_FILES",
            "percentComplete": 0
        },
        "uploadLocation": {
            "bucket": "assets.cesium.com",
            "endpoint": "https://s3.amazonaws.com/",
            "prefix": "sources/1207176/",
            "accessKey": "key",
            "secretAccessKey": "key",
            "sessionToken": "token"
        },
        "onComplete": {
            "method": "POST",
            "url": "https://api.cesium.com/v1/assets/1207176/uploadComplete",
            "fields": {}
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer key"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"}}"
    },
    "request": {}
}

As you can see, there is a onComplete object, with the url ‘https://api.cesium.com/v1/assets/‘+id+’/uploadComplete’ that you can then use to tell ion the upload is Complete.
but when I use from (so when I create the asset on ion, and tell him I will upload on my own bucket) this is what I get :

{
    "data": {
        "assetMetadata": {
            "id": 1205235,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-13T20:45:13.364Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer key"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"bucket\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"key\",\"secretAccessKey\":\"key\"}}}"
    },
    "request": {}
}

As you can see, there is no onComplete.url. There is no onComplete object at all.
So my question is, how do you tell ion, you have finished uploading on your own bucket, start tilling please ?

Since you are importing from S3, there is no onComplete as there is no “upload”. When importing from S3, the data is direct sent to the tiling process from your S3 bucket. So the next step after post is to check for progress.

Hi @Shehzan_Mohammed ,

Thanks, that make perfect sense. So my mistake was not to include the keys when creating the assets.
Now, it’s kind of working, but I have to figure out why it start tilling but then stop, & tell me a problem was detected, only when I use the rest api.
When I create the asset on the assets page, with the same file in my S3 bucket, the tilling work fine.