Avoid download and upload of files by using the S3 SDK Copy functionalities

Cesium ion
1

Hello,

My question is mostly aimed at reducing the network time and egress costs by downloading big files from our bucket and uploading them to Cesium as assets for tiling.

Our files are stored in Wasabi, an S3 provider different from Aws which offers free egress (given terms and conditions). Cesium allows to upload files from an S3 bucket which would work perfectly if Wasabi S3 was supported alongside Aws S3. Since Wasabi free egress would mean that we have to pay no cost when Cesium will read the assets from the Wasabi bucket and we avoid a server roundtrip which would add more time to the whole operation.

Unfortunately only Aws S3 is supported so we are looking at alternative ways and, aside from Self-Hosted which we are not ready for, the only other possibility we could think of is to leverage the S3 SDK Copy functionalities. They allow an S3 bucket to send data directly to another S3 bucket without the data having to go through an intermediary.

The question is if this is possible with Cesium Ion and, if so, how could we make it happen given the need to configure cross account permissions on both destination and source buckets.

Thank you in advance for the help!

2

Hi,

You can utilize the Cesium ion REST API to do what you stated above. As part of the REST API, you will first need to make a POST request with your required options. The Cesium ion API server will return a location to you that contains a bucket and a prefix along with temporary credentials for the destination bucket. You can then use that to upload your source data. Once you are done, you can make another request to Cesium ion’s REST API stating that your upload is complete.

Read more about the API documentation Ion REST API documentation – Cesium
This blog post is a little bit older but it breaks down the steps I listed above Upload with REST – Cesium

Thanks,
Ankit

3

Hi Ankit,

Thank you for your quick response!

I am not sure if I am getting what you are saying but it sounds like I can use the Copy functionalities of the S3 SDK with the temporary credentials provided by Cesium when initiating an asset upload. I will be trying that today and report back here on how it goes :+1:

4

I have tried to simply use the Upload with REST approach and couple it with the CopyPart method of the SDK, but since the source bucket is in Wasabi and the Destination bucket is owned by Cesium (and likely in Aws S3), the SDK cannot be initialised with a set of credentials that satisfies both.

From a deeper search there is nothing that indicates SDK support for copying files between two different storage providers so I guess what we are trying to do is not possible and we will have to download the file first from Wasabi and then upload it to the bucket provided by cesium :pensive_face: