Connect cesium ion to own S3 bucket with REST API

romza · July 9, 2022, 1:58pm

Hi,
I’m trying to use the REST API to create an asset to cesium ion, then upload the file to my bucket, then tell cesium ion where the file is so it can do the tilling.
If I understand correctly, I have to tell on the first step, when I create the asset in cesium ion, that the file is hosted on my bucket, using the “from” object.
My problem is, I can only create the asset if I don’t have the from part.
This is not working (Request failed with status code 400):

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: '3DTILES',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      },
      from: {
        type:'S3',
        bucket: 'mybucket',
        credentials :{
          accessKey:'key',
          secretAccessKey:'secretkey'
        }
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

This is working :

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: '3DTILES',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

but then when I want to tell ion the file has been uploaded to S3, I get an error, because he doesn’t understand the file is in mybucket, not cesium bucket.
What am I doing wrong with from ?
Thanks !

Shehzan_Mohammed · July 10, 2022, 6:52pm

I don’t see the prefixes listed in your from object. See the example on the right hand side of Ion REST API documentation – Cesium.

romza · July 11, 2022, 8:31am

Hi @Shehzan_Mohammed,
Ok, so I have tried like this :

axios.post('https://api.cesium.com/v1/assets', {
      name: selectedFile.name,
      description: '',
      type: 'POINT_CLOUD',
      options: {
        position:[ 2.29, 48.85, 0.1],
        sourceType: srcType
      },
      from: {
        type:'S3',
        bucket: 'my',
        prefixes: ['myprefix'],
        credentials :{
          accessKey:'mykey',
          secretAccessKey:'mysecretkey'
        }
      }
    },{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => assetCreated(response))
    .catch(function (error) {
      console.log(error);
    });

Now I get a 409 error.

romza · July 11, 2022, 9:33am

Sorry, my bad. it’s working with prefixes. Thanks !

romza · July 11, 2022, 10:33am

Hi @Shehzan_Mohammed,

Ok so now I create my assets, upload it to s3, but when I want to tell ion that it’s done, I get a 409.
I’m pretty close, I can feel it, but not there yet

This is what is answered when I create the asset :

{
    "data": {
        "assetMetadata": {
            "id": 1198055,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-11T09:43:23.636Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-length": "184",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer mytoken"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"mybucket\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"secret\",\"secretAccessKey\":\"mysecret\"}}}"
    },
    "request": {}
}

So now I try to tell ion the upload is finished like this :

var body = onComplete.config.data;
    axios.post(onComplete.config.url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });

But then I get a 409.
I guess there is something wrong with the body var, but I can’t figure out how it should be.
Thanks for the help !

Shehzan_Mohammed · July 11, 2022, 1:03pm

Have you taken a look at cesium-ion-rest-api-examples/index.js at main · CesiumGS/cesium-ion-rest-api-examples · GitHub?

romza · July 11, 2022, 2:50pm

Yes, but when I look at the answer from the call that create the assets in ion, when I use from, I don’t have the onComplete object.
Here is what I get :

{
    "data": {
        "assetMetadata": {
            "id": 1198632,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-11T14:40:31.257Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-length": "184",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer mybearer"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"mybucket\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"mykey\",\"secretAccessKey\":\"mysecretacceskey\"}}}"
    },
    "request": {}
}

When I don’t use “from” in the first call, I get a response with an “onComplete” object like in the example you give, with an url.
Here it’s missing.

Shehzan_Mohammed · July 13, 2022, 5:24pm

A 409 requests indicates some type of conflict. From your original comment in Connect cesium ion to own S3 bucket with REST API - #3 by romza, here are the issues I see:

type should be 3DTILES. The type indicates the type of output, not input. Use the options.sourceType as POINT_CLOUD instead.
Regarding options.position - if your point cloud already has a CRS embedded, then this is not needed.

These are the ones visible to me. You may need to double check your AWS keys, but I can’t confirm that from my end. I would recommend using the Web browser to replicate this upload using the same options (keys, buckets, prefix etc), ensure that upload is successful, and then using the CLI.

romza · July 14, 2022, 10:29am

Hi @Shehzan_Mohammed ,
Thanks for the help, you were right about the type, it should be 3DTILES.
But my problem is not to upload the file to my bucket, this is working, so there is no issue with my AWS keys.
It’s when the upload is done, I’m missing the url to tell ion it can start tilling.
I tried :

    var body = onComplete.config.data;
     axios.post(onComplete.config.url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });

Since I’m missing the onComplete url, I tried with the config.url (https://api.cesium.com/v1/assets)
But I get a

POST https://api.cesium.com/v1/assets 409

So I tried to recreate the onComplete url like this :

var id = onComplete.data.assetMetadata.id;
var url ='https://api.cesium.com/v1/assets/'+id+'/uploadComplete';
var body = onComplete.config.data;
axios.post(url, body,{
      headers: { Authorization: `Bearer ${accessToken}` }
    })
    .then(response => monitorTilling(response))
    .catch(function (error) {
      console.log(error);
    });

but then I get a

POST https://api.cesium.com/v1/assets/1207043/uploadComplete 400

My problem is that when I post to https://api.cesium.com/v1/assets, if I don’t use from, I get in return this :

{
    "data": {
        "assetMetadata": {
            "id": 1207176,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-14T10:19:01.128Z",
            "status": "AWAITING_FILES",
            "percentComplete": 0
        },
        "uploadLocation": {
            "bucket": "assets.cesium.com",
            "endpoint": "https://s3.amazonaws.com/",
            "prefix": "sources/1207176/",
            "accessKey": "key",
            "secretAccessKey": "key",
            "sessionToken": "token"
        },
        "onComplete": {
            "method": "POST",
            "url": "https://api.cesium.com/v1/assets/1207176/uploadComplete",
            "fields": {}
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer key"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"}}"
    },
    "request": {}
}

As you can see, there is a onComplete object, with the url ‘https://api.cesium.com/v1/assets/‘+id+’/uploadComplete’ that you can then use to tell ion the upload is Complete.
but when I use from (so when I create the asset on ion, and tell him I will upload on my own bucket) this is what I get :

{
    "data": {
        "assetMetadata": {
            "id": 1205235,
            "type": "3DTILES",
            "name": "House (1).laz",
            "description": "",
            "bytes": 0,
            "attribution": "",
            "dateAdded": "2022-07-13T20:45:13.364Z",
            "status": "NOT_STARTED",
            "percentComplete": 0
        }
    },
    "status": 200,
    "statusText": "",
    "headers": {
        "cache-control": "no-cache",
        "content-type": "application/json"
    },
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "Bearer key"
        },
        "method": "post",
        "url": "https://api.cesium.com/v1/assets",
        "data": "{\"name\":\"House (1).laz\",\"description\":\"\",\"type\":\"3DTILES\",\"options\":{\"position\":[2.29,48.85,0.1],\"sourceType\":\"POINT_CLOUD\"},\"from\":{\"type\":\"S3\",\"bucket\":\"bucket\",\"prefixes\":[\"myprefix/prefix\"],\"credentials\":{\"accessKey\":\"key\",\"secretAccessKey\":\"key\"}}}"
    },
    "request": {}
}

As you can see, there is no onComplete.url. There is no onComplete object at all.
So my question is, how do you tell ion, you have finished uploading on your own bucket, start tilling please ?

Shehzan_Mohammed · July 14, 2022, 12:59pm

Since you are importing from S3, there is no onComplete as there is no “upload”. When importing from S3, the data is direct sent to the tiling process from your S3 bucket. So the next step after post is to check for progress.

romza · July 15, 2022, 4:15pm

Hi @Shehzan_Mohammed ,

Thanks, that make perfect sense. So my mistake was not to include the keys when creating the assets.
Now, it’s kind of working, but I have to figure out why it start tilling but then stop, & tell me a problem was detected, only when I use the rest api.
When I create the asset on the assets page, with the same file in my S3 bucket, the tilling work fine.

romza · July 25, 2023, 10:23am

Hi @Shehzan_Mohammed, for SEO reason, my client ask me to delete this post. I cannot find a way to do that.
Would it be possible to delete it or to de index this post ?
Thanks,
Bets regards,

Topic		Replies	Views
Issues when using "create new asset with AssetFrom " API request Cesium ion ion-upload	5	614	December 21, 2023
How to connect cesium ion to my bucket s3? Cesium ion ion-upload	11	2235	July 7, 2022
REST API upload to S3 access denied Cesium ion	4	440	July 11, 2022
Is there any way to upload cesium ion assets using javascript application to cesium ion account? Cesium ion ion-upload , rest-api	11	999	April 29, 2019
Amazon S3 Help! CesiumJS	2	498	June 14, 2021

Connect cesium ion to own S3 bucket with REST API

Related topics