Programmatic Access to Cloud Optimized GeoTIFF (COG) Global Datasets
Leveraging the Power of COGs
OpenTopography strives to provide a variety of access methods to our datasets to address the varying needs of our users. While most users access data through the OpenTopography web map interface, others prefer to programmatically access data from our cloud-based bulk storage.
To facilitate programmatic access to cloud-based data, we have recently converted our entire catalog of global topographic data to Cloud Optimized GeoTIFFs (COGs). COGs are quickly becoming a standard format for hosting raster data in the cloud due to the optimized file structure to enable faster access and subsetting. With the COG format, programs and web requests can read the top-level header and quickly extract data for a region of interest as opposed to having to read the entire file. This drastically reduces file read-times as well as reducing the number of http GET requests to access the data.
We've created a Jupyter Notebook to demonstrate different methods of accessing data from OpenTopography's cloud-based bulk storage using the NASA Shuttle Radar Topography Mission (SRTM) dataset as an example. The notebook illustrates how to use open-source, command line tools to programmatically access data without the need for a web-map interface. In addition, the notebook highlights the power of Cloud Optimized Geotiffs (COGs) and how they can be used to reduce download file sizes as well as increase data access speeds.
Explore the notebook for more information on:
- Benefits of using Cloud Optimized GeoTIFFs (COGs)
- Using GDAL to access COGs
- Using GDAL vector tools
- Reading COGs and vector data with Python libraries
Access the notebook by cloning our github repo here: OpenTopography Jupyter Notebooks
To run the notebook interactively: Access the notebook with Binder.
The notebook walks through a simple workflow using SRTM topography for the Grand Canyon in Arizona. Users will access the SRTM COGs via OpenTopography cloud storage; subset and mosaic the data; download, clip, and buffer a vector file of the Colorado River; and then clip the DEM to the buffered area. Below is a visualization of some of the steps users will work though as part of the Jupyter Notebook.
Accessing COGs from OpenTopography's Bulk Storage using AWS Command Line Tools:
Using GDAL vsicurl with OpenTopography COGs to subset and mosaic the data:
Download, clip, and buffer Colorado River boundary to area of interest:
Use Rasterio to read COGs directly to polygon boundary:
Notebook Citation:
Beckley, M., Crosby, C., Nandigam, V. & Phan, M. (2021). Programmatic Access to OpenTopography’s Cloud Optimized GeoTIFF (COG) Global Datasets (Version v1.0). https://doi.org/10.5069/G9BC3WRK Accessed: MM/DD/YYYY