Hi all,
Are there currently any public solutions allowing for 3DTile datasets to be packed into a single large file, rather than many (many!) small ones?
Large scale 3DTile projects produce millions of tiny files, which is incredibly inefficient from a data management perspective, as most systems don’t deal well with it. The smaller the files are, the greater the overhead per file, e.g. header to payload, waste from byte to block alignment, cost/time per file op vs per MB, etc.
One example is simply copying or uploading a folder of 3DTiles hierarchy, which might take an hour as small files (Windows often shows just a few MB per second, even local SSD-to-SSD) versus a few minutes for a single large combined file (Windows will usually tap out at 300MB/s+).
Another example is that Amazon’s S3 storage has various costs per file, and you can’t get the benefit from some of the cost saving mechanisms with small files (e.g. Intelligent Tiering only works for files 128kb or larger).
I can see two options:
-
3DTiles format itself supports a single large BLOB to be indexed into. This is preferable as it keeps it simple from the user perspective (i.e. no custom file server required); or
-
3DTiles deals only with a hierarchy of files as it does now, but a file server component receives those requests and retrieves them from the single large BLOB (i.e. https://a/b/c/d.json is actually being extracted out of https://a/b.zip and returned as if it was a separate file)
I suspect all the companies dealing with very large scale datasets already have a proprietary solution in place for this at the file server level, but seems like it would benefit the greater good if it was part of the base spec.
Keen to hear any thoughts on this, as the size of these datasets is only going to keep getting bigger – perhaps exponentially so!
Regards,
Andrew