Cloud Filter the Cloud-native Way: Planet UDM2 in Google Earth Engine
In the year 2010, a paper by Roy et al. discussed the mechanics of creating cloud-free Landsat composites over monthly, seasonal, and yearly time periods. The project was called Web Enabled Landsat Data (WELD) and was discontinued in 2019.
Since the Landsat Program was founded nearly a half-century ago, the number of optical sensors, instruments, and satellites have made huge leaps and bounds in their capabilities. In around 2011–2013, Google Earth Engine, a cloud computing platform, allowed users to quickly generate these cloud-free composites with varying degrees of control in a matter of seconds. This represented a huge move forward in the capability of cloud-native platforms and datasets.
It led to the creation of the first EarthEngine-generated Landsat browsable time lapse video.
This idea of looking at cloud and cloud-free composites became interesting, as it could allow for imaging quality to be more predictable. Not only could you track the clouds instead of getting rid of them, but you could also determine areas where multiple optical datasets could help. For example, Wilson and Jetz’s 2016 paper on the Global Cloud Study was a step in mapping that dynamics from twice-daily MODIS datasets.
Over the last week, I’ve seen two articles on machine learning generated cloud detection and masking, which include improvements to cloud detection capabilities using ML models for Sentinel-2 and a platform-specific method. Since this is a common thread in such discussions, I thought the time was right to discuss Planet’s Machine Learning-Based Usable Data Mask V2, also known as UDM2. It’s something we have been consistently producing since August 2018. In the past, my colleague Ben wrote a great article introducing the idea of what is usable data in our imagery. Read more about UDM2, including the methodology used for the generation, here.
Cloud Utilities and Google Earth Engine
The objective of this tool and methodology is very simple. How do you filter not just a scene by an AOI based on the cloud, cloud shadow, haze, and other band information that UDM2 provides?
This tutorial has the following assumption
- We ingest UDM2 images into Google Earth Engine (in this case, a collection is made available for use).
- The image metadata is retained on the UDM2 images to make filtering this easier.
- An active Google Earth Engine account is needed. (You can sign up for one here.)
Cloud-Utils Module
Cloud utils are tools to work with Planet’s UDM2 dataset. The Usable Data Mask v2 is an 8 bit 8 band image with the following bands.
These tools are designed as modules that can be imported directly into any Google Earth Engine script and can then be used in a plug-and-play manner. UDM2 classification stems from a supervised machine learning technique and is designed to separate the above classes. Things to note:
- All classes are mutually exclusive, meaning you can be either of the classes in Band-1 to Band -6.
- The classes (Band-1 to Band-6) are binary, as mentioned in the band list.
- The confidence map value, therefore, refers to a single class and has a 1:1 correlation to the underlying imagery.
To read more about the UDM2 Classification Methodology, you can go here to our Developers Center.
This module and the example files are both bundled in the same repository. You can accept the repository and it should be available with read privileges. Make sure you have a Google Earth Engine account, and if not, set one up using the guide here. This code, repo, and assets are plug-and-play and can be applied to your own Planet imagery and UDM2 values. You can find the setup for a Planet account here.
Getting Started
You can click here and go to the GitHub page for a copy of the Earth Engine code or
Accept the Google Earth Engine Repo Here
Composite Tools
Planet serves up daily data and as such, daily composite was the focus of this module. The composite tools which are part of this module are general tools and can be utilized for image collections that have the system:time_start characteristic. This ensures you can do the following, and you can do this easily with any collection (for example daily MODIS or VIIRS collections).
Dynamic Cloud Filter Export
The dynamic cloud filter and export tool is designed to allow the user to calculate total percentage stats for all bands in the UDM2 imagery. Meaning for each AOI, you get a percentage clear pixel, snow cover, shadow, and so on. The method is simply based on summarizing these values over the overlapped AOI and then exporting these percentages up to two decimal places. The method also takes into account that because the daily composite or imagery may not overlap with the entire AOI, it also generates a percentage overlap. The example runs this on all imagery in the collection, as well as daily composites.
The example then chooses the collected metrics and exports them as CSV files. The script uses a helper function to generate the PlanetScope 4-Band imagery for which these UDM2 files are generated, so they can then be directly used to order imagery.
The exported CSV reports are easy to use to then filter out by dates or specific images in the collection, which could then be used to order imagery.
Cloud Masking
But what if you have the images and then you want to mask? You can do that too just like the WELD project and just like the Landsat Cloud masking. I look forward to updated utils for masking. Adding the PlanetScope 4-band cloud masking below.
I hope you enjoy this project, as we learn how to build around the idea of understanding cloud dynamics, cloud predicted tasking and cloud masking to collectively improve capabilities. If you find this guide and tools useful please or if you have used it to create a better workflow, let me know.
