Archived Training Dataset Downloads now Available on Radiant MLHub.

Kevin Booth
Feb 8 · 3 min read

A little over a year ago, we launched the first iteration of Radiant MLHub in the form of a STAC-compliant API, which allows you to browse our training data collections and list and download individual assets from the items within those collections. Today, we’re announcing the ability to download an archived version of training datasets with just a single-click download. In this post, we’ll describe the process for downloading datasets, the structure of the archived datasets, and provide some tips for effectively traversing the downloaded datasets.

Image for post
Image for post
Building Footprint Labels, Tropical Storm Imagery and Wind Speed Labels, and Crop Type Labels all available on Radiant MLHub.

Downloading Datasets

We are now offering three different methods of downloading our datasets. The easiest method, downloading on our registry, can be accessed by navigating to a dataset page and clicking on the “Download” link for each collection you would like to download. Clicking this link will direct you to our dashboard, which will ask you to login if you are not already authenticated and then begin the download process for that collection.

The second easiest method is downloading datasets via our new archive API endpoint. By making a GET request to the endpoint below with the ID of the collection you’d like to download replacing “{COLLECTION_ID}” and your API key replacing “{API_KEY},” an HTTP status code 302 response will be returned with the location of the archive file located within the “Location” header in the response.

The final and most advanced method of downloading datasets would be to navigate our STAC API directly and download each individual STAC item. We do not recommend this if you intend to download the entire dataset.

Archive Structure

Once extracted, each archive download contains a similar structure. Files that are common to all items (i.e., documentation) are contained within the “_common” folder. Each STAC item then has its own folder, and contained within these folders are all of the assets for that item and a “stac.json” file which contains the STAC metadata for that item.

An example of the “ref_landcovernet_v1_labels” and “ref_landcovernet_v1_source” collections downloaded and extracted into the same parent “archives” folder is shown below.

Tips

Some datasets contain multiple label types or multiple source imagery types. For example, our Chesapeake Land Cover dataset has land cover labels, NLCD labels, building footprint labels, Landsat source imagery, and NAIP source imagery. If you are only interested in the NLCD labels and NAIP source imagery, then you only need to download the NLCD and NAIP collections.

The item STAC metadata contains relative links to all of the assets which are related to that item within the “assets” key. For label items, the STAC metadata will also contain relative links to all source imagery STAC files which are related to that label item. These relative links all assume that both collections are extracted into the same parent folder, as shown in the file structure example above.

Conclusion

The addition of these new methods of downloading datasets should eliminate the frustration of having dataset downloads, which took multiple days to complete and, in some cases, failed in the middle of a download. We are constantly looking for ways to make datasets more accessible and easy to integrate with your processes. If you have any suggestions on improvements we can make to Radiant MLHub, feel free to send us an email at hello@radiant.earth

Radiant Earth Insights

Earth Imagery for Impact

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store