2019-12-30 / Rob Spearman / Terrain
This blog post is a little different than usual to give you a peek behind the scenes at some of the trials and tribulations processing data for the dome.
A few months ago we released version 2.0 of our add-on terrain datasets: Solar System Collection and Whole Earth. This new generation is fully pre-tiled for immediate startup and fast loading. All data is stored on solid state drives (SSD) for maximum performance. (For customers with our 1.0 datasets we highly recommend a free data refresh to 2.0.)
Because the data is local to your planetarium system you do not have to fly around to pre-cache anything, and will not be held up by network speeds or server problems. Portable planetariums can fully utilize terrain offline. Another benefit is not having to pay continually for server access.
The Whole Earth dataset includes imagery down to 15 meters per pixel (mpp), along with high resolution topography, night lights, clouds, and a water map (for reflections). Using the dataset is really just as simple as plugging in a drive and flying where you want to go with a wireless gamepad controller.
Above is a short dome-view video of a user flying live around Washington State, past Mount Rainier, and ending above our hometown of Bremerton.
We often push the limits of common tools due to the huge datasets we are processing. Often times the tools work well, but only for small datasets. The developers just never considered needing to process an entire planet, or the whole sky rather than just a city or a nebula.
Converting just a single 15mpp Earth imagery layer was going to take more than a year with the standard tool we were using. The estimated time to completion was actually only displayed in seconds, so you can imagine how out of the expected we were to the tool creator. We had to creatively parallelize the process and move off SSDs, which you might be surprised to hear were just too slow.The most common SSDs still use the SATA III interface. This was a heavily I/O intensive process, so raw read/write speed was the paramount issue for us. NVMe uses a much faster PCI-express interface. Here is a comparision of bandwidth:
|SATA III mechanical||~100 MB/s|
|SATA III SSD||~500 MB/s|
|NVMe SSD||~3500 / 3000 MB/s|
In the end we got this single layer processed in about a month using an NVMe drive. The resulting tile sets are huge, almost 4TB in size.
The takeaway? Nothing is easy with big data, and dataset sizes continue to grow and tax processing tools and hardware.