New collection of Jupyter Notebooks enables programmatic access to cloud-hosted USGS 3D Elevation Program (3DEP) lidar data
by Cole Speed, 2023
The U.S. Geological Survey 3D Elevation Program (3DEP) is systematically acquiring and facilitating public access to high-quality light detection and ranging (lidar) data spanning the conterminous United States, Hawaii, and US territories over an 8-year period (2015-2023). Upon completion, 3DEP lidar data will provide the first-ever national baseline of consistent high-resolution elevation data – both 3D point cloud and bare earth – and will be instrumental in addressing the rapidly-growing need for accurate 3D representations of landscape topography, vegetation distribution and structure, and infrastructure across the United States.
Green polygons indicate the coverage of 3DEP lidar data currently (as of 11/16/2023) available for download. For up-to-date lidar data coverage, including "in-progress" datasets, see the USGS 3DEP website.
As of September 2023 approximately 94% of the United States landmass has high-resolution (meeting 3DEP quality specifications) elevation data available or in-progress, including over 2000 unique lidar datasets amounting to more than 57 trillion individual points. 3DEP lidar data are freely available and hosted in an Amazon Web Services public S3 bucket as cloud-native Entwine Point Tiles (EPT), a lossless, streamable, octree format based on LASzip (LAZ) encoding. While access to 3DEP point cloud data and tools for generating derivative products (e.g., digital elevation models and a variety of other raster products) are currently provided via several seamless user interfaces including the OpenTopography Data Portal and the USGS National Map, programmatic access to process and visualize these data are underdeveloped. Programmatic access may be preferred by users seeking to employ tailored processing pipelines designed for their specific use-cases and power users seeking to access and process immense swaths of 3DEP lidar point cloud data on local, commercial cloud, or high performance compute infrastructure. Therefore, development of these programmatic workflows for 3DEP data will further data utilization and motivate additional creative uses for these already substantial – and growing – cloud-hosted resources.
OpenTopography, supported by the USGS Community for Data Integration (CDI), has developed a new suite of Jupyter Notebooks designed to enable access and enhance usability of USGS 3DEP lidar data through easy-to-use programmatic workflows. Jupyter Notebooks combine live code, documentation, and visualizations in a single document, facilitating reproducible data access and processing workflows. The notebooks include customized functionality leveraging cloud resources, USGS Web Map Service APIs, and open-source Python libraries. Each workflow is tailored for a specific use-case and includes custom functions for accessing, processing, visualizing, and downloading point cloud and raster products for user-defined areas of interest (AOIs). Point cloud data access and processing is achieved via the Point Data Abstraction Library (PDAL) through customized “pipelines” used to define file input/output and intermediate processing steps. AOIs are specified within the notebooks using either (1) a user-defined bounding box drawn on an interactive ipyleaflet map; (2) a user-input vector polygon file; or (3) a boundary obtained via a USGS Web Map Service API. Point cloud data within the AOI are retrieved from the AWS public bucket and subsequently processed according to user specification. For simplicity, the workflows employ custom Jupyter widgets in specifying input parameters to data access and processing functions.
Above are example portions of a Jupyter Notebook designed to generate a digital terrain model for a segment of the Brazos River, Texas.
Example of an interactive ipyleaflet map within a Jupyter Notebook that allows users to select an area of interest where 3DEP data are available.
To further enhance access and ease-of-use, this suite of Jupyter Notebooks have been designed to run on the Google Colaboratory cloud platform (Colab, for short), in addition to the user's local compute infrastructure. Running the notebooks in Google Colab creates a virtual environment and installs dependencies on Google's cloud servers, thereby eliminating software dependency issues caused by differences in a user's local operating system, package and environment manager, and versions of specific Python libraries. An additional benefit to running the notebooks on Google Colab is direct access to Google's high-end CPU/GPUs and 12GB ram allotment per runtime (upgradable to 25GB with a Colab Pro subscription).
Users may also choose to download and run the notebooks locally. The notebooks, documentation regarding dependency installation, and additional resources are available in the OpenTopography USGS 3DEP Jupyter Notebook Github Repository and in the section below.
While the Jupyter Notebooks have been designed to enhance access and promote creative and impactful uses of 3DEP data, we also hope they will be useful as educational tools related to general Python programming, high-resolution topographic data, and accessing/using cloud-hosted data resources. To this end, detailed plain-language documentation within each notebook provides instruction for execution and customization, making them approachable for users with less experience in the Python programming language or for those only looking to learn about high-resolution 3D topographic data and applications. Community feedback regarding usability, suggested feature enhancements, and specific use-cases is requested and encouraged. Feedback may be provide via email to info@opentopography.org or by submitting an issue to the OpenTopography USGS 3DEP Jupyter Notebook Github Repository.
Available Notebooks
All of the Jupyter Notebooks in this suite are designed to enable access to cloud-hosted USGS 3DEP lidar data for a user-defined areas of interest, perform customized processing, and generate derivative products (e.g., digital elevation models, canopy height models, topographic difference models, and a few others). Below are descriptions of the available Jupyter Notebooks, links to view their contents and execute them in Google Colaboratory, and illustrations of deliverables produced using these Jupyter Notebook-based programmatic workflows. Notebooks may be launched via Google Colab in the browser by clicking the "Open in Colab" widget associated with each notebook. It is advised to keep point cloud jobs smaller than ~200,000,000 points when running on Colab servers, as processing of larger point clouds may exceed the Google Colaboratory 12GB ram allotment.
1. Generate point cloud and DTM/DSM for a user-defined area of interest
This notebook represents the basic functionality built into all of the other, more specific, notebooks. The user may either select an area of interest using an interactive map or import a vector shapefile polygon that defines the region of interest. Point cloud data within the polygon are retreived, and a digital surface model (DSM) or digital terrain model (DTM) may be subsequently generated.
(Upper) Selection of a region over downtown Chicago, IL in an interactive ipyleaflet map. (Lower) Point cloud and digital surface model (DSM) for the region selected in the upper panel.
2. Generate point cloud and DTM/DSM for USGS boundaries (7.5 x 7.5' quadrangles and HUCs)
Two notebooks have been developed to enable users to retrieve and process point cloud data corresponding to two USGS-maintained dataset boundaries, namely the (i) 7.5 x 7.5' USGS Quadrangles and/or (ii) USGS 12- or 14-digit hydrologic unit code (HUC) watershed boundaries.
(i) The USGS 7.5 x 7.5 ' quadrangle boundary notebook allows users to select a point location (or input a coordinate pair) and retrieve point cloud data from within the 7.5' quadrangle. A digital surface model (DSM) or digital terrain model (DTM) for the quadrangle may then be produced. The notebook relies on the USGS Topo map service API for obtaining the corresponding 7.5' quadrangle boundary.
(ii) The USGS 12- and 14-digit hydrologic unit code (HUC) boundary notebook allows users to select a point location (or input a coordinate pair) and retrieve point cloud data from within the corresponding 12- or 14-digit HUC watershed. A digital surface model (DSM) or digital terrain model for the watershed may then be produced. The notebook relies on the USGS WBD map service API for obtaining the corresponding watershed boundary.
View the USGS 7.5' x 7.5' Quadrangle notebook or 
View the USGS Hydrologic Unit Code notebook or
(Left) Digital elevation model produced for the Cross Canyon 7.5-minute USGS Quadrangle shows the southwest-northeast trending grabens of the Needles Fault Zone in Canyonlands National Park, Utah. (Right) Digital elevation model for the Pimmit Run-Potomac River subwatershed (12-digit HUC 020700100103), Washington D.C.
3. Generate point cloud and DTM/DSM for a buffered corridor region
This notebook allows the user to draw a line feature on an interactive map or import a line vector shapefile defining the corridor of interest (e.g., river centerlines, road/highway paths, utility line locations, etc). The feature is buffered based on user specifications, point cloud data are retreived within the polygon, and DSM/DTMs may be generated for the region.
View the corridors notebook or
3DEP lidar point cloud for a segment of the Brazos River, Texas and surrounding floodplain. The corridor is defined by a buffered user-input river centerline.
3DEP lidar point cloud data for a buffered corridor encompassing a segment of electric power transmission lines. Corridor boundary is a segment from the Homeland Infrastructure Foundation Level Database (HIFLD). Visualized in CloudCompare.
4. Vertical differencing: Generate Canopy Height Model (CHM) and time-lapse topographic difference models
Two notebooks have been developed to produce vertically differenced products for a user-defined area of interest.
(i) The Canopy Height Model (CHM) notebook allows the user to select an area of interest using an interactive map or import a vector shapefile polygon. A digital surface model (including trees and constructed features) and a digital terrain model (representing bare earth topography) are generated and differenced, producing a representation of tree height or height of constructed features within the area of interest.
View the canopy height model (CHM) notebook or
(ii) The time-lapse topographic differencing notebook allows the user to select an area of interest using an interactive map or import a vector shapefile polygon. The region must have co-located 3DEP surveys collected at different times. A digital terrain model (representing bare earth topography) is generated for each time period and then differenced to produce a model of landscape topographic change between the 3DEP acquisition periods.
View the topographic differencing notebook or
(Left) Canopy height model for a region of Great Smokey Mountains National Park produced by differencing co-located digital surface and digital terrain models. Colorbar indicates tree canopy height. (Right) Time-lapse topographic difference of the Wax Lake Delta, southern Lousiana. Colorbar indicates erosion and deposition on the delta between 2011-2013.
5. Produce colorized 3DEP point clouds
This notebook allows the user to produce a point cloud that has R,G,B values as a point attribute. The user may either select an area of interest using an interactive map or import a vector shapefile polygon. Point cloud data within the polygon are retreived for the AOI and co-located imagery from the National Agriculture Imagery Program (NAIP) is also obtained for the region from the USGS NAIP Plus map service. The PDAL **filters.colorization** function assigns the points the corresponding R,G,B values as point attributes, returning a point cloud that can be visualized in true-color.
View the colorize point clouds notebook or
Lidar point cloud covering the University of Texas at Austin campus (Austin, Texas). (Right) Point cloud data colored by elevation. (Left) Point cloud data colored by RGB values from co-located National Agriculture Program (NAIP) 1-meter imagery. Visualized in CloudCompare.
Funding Acknowledgements and Call for User Input
This suite of Jupyter Notebooks was initially developed during the summer 2022 by Cole Speed, with support from OpenTopography, the USGS Community for Data Integration, and the UNAVCO Student Internship Program. The notebooks will continue to be expanded and adapted based on user input and feedback, which may be provided via email to info@opentopography.org or submitting an issue to the OpenTopography USGS 3DEP Jupyter Notebook Github Repository.