All Clear with Cloud Score+

By Valerie Pasquarella, Research Scientist, Geo Sustainability

At Geo for Good 2023, we announced the launch of Cloud Score+, our solution to the Sentinel-2 cloud and cloud shadow detection problem. If you’ve ever worked with optical satellite imagery, you’re probably very familiar with clouds and their shadows. Clouds intersect incoming solar illumination, projecting their shadows onto the landscape below.

Occlusion effects of clouds and cloud shadows

In some cases, clouds are thick and completely block light being reflected from the surface. In other cases, clouds are high and thin, only partially occluding surface conditions and casting wispy shadows over large areas. At the end of the day, most analysts like you just want to remove the clouds and their shadows to work with the clearest, best available pixels. That being said, everyone seems to have their own definition of what is “usable” or “clear”.

Rather than working backwards from “what is a cloud?”, Cloud Score+ was developed with a focus on improving the overall quality assessment experience. Cloud Score+ solves the challenge of identifying clear observations by providing a comprehensive per-pixel “usability” score that can be used to mask or simply weight observations based on their overall quality. Cloud Score+ results for the Copernicus Programme’s Sentinel-2 mission are available now as a public Earth Engine Image Collection. This collection is being operationally generated for all newly ingested Sentinel-2 acquisitions. We’ve already completed global backfill through January 2021, and backfill for the rest of the Sentinel-2 collection is currently underway (see the Updates section below for the latest status or browse the collection using our Cloud Score+ Collection Explorer).

Because using Cloud Score+ requires bringing together the Cloud Score+ and Sentinel-2 image collections, we are also launching a new Earth Engine algorithm, linkCollection(). This convenience function makes it faster and easier than ever to add bands from separate images or image collections based on a shared image property. With just a single line of code, you can link Cloud Score+ QA bands to the corresponding Sentinel-2 L1C or L2A image or image collection and proceed with your analysis.

Cloud Score+ provides a continuous QA score that can be thresholded to mask clouds, cloud shadows, and other atmospheric occlusions.

Get started with Cloud Score+

Let’s say you want to use Cloud Score+ to create a clear median composite. First, start with a collection of Sentinel-2 top-of-atmosphere (L1C) or surface reflectance (L2A) imagery and filter by date and region of interest. Next, use linkCollection() to append the Cloud Score+ bands from the Cloud Score+ Image Collection and map a function that applies a binary mask to each image based on a thresholded Cloud Score+ QA value. Then just apply a median() reducer to get a clear composite. Not happy with the results? Adjust the threshold. That’s it:

var s2 = ee.ImageCollection('COPERNICUS/S2_SR_HARMONIZED');
var csPlus = ee.ImageCollection('GOOGLE/CLOUD_SCORE_PLUS/V1/S2_HARMONIZED');

var ROI = ee.Geometry.Point(-119.9087, 37.4159);

var QA_BAND = 'cs';
var CLEAR_THRESHOLD = 0.60;

var composite = s2
    .filterBounds(ROI)
    .filterDate('2023-01-01', '2023-02-01')
    .linkCollection(csPlus, [QA_BAND])
    .map(function(img) {
      return img.updateMask(img.select(QA_BAND).gte(CLEAR_THRESHOLD));
    })
    .median();

var s2Viz = {bands: ['B4', 'B3', 'B2'], min: 0, max: 2500};

Map.addLayer(composite, s2Viz, 'median composite');
Map.centerObject(ROI, 11);

🔗 View this code in the Code Editor

No more bit shifts. No more multi-step cloud and shadow detection. Just a single continuous score… well, really two scores… cs and cs_cdf. For both scores, values closer to 1 are considered to be closer to clear reference conditions and therefore more usable, while scores approaching 0 are deemed to be partially or fully occluded and therefore less usable. You can think of cs as a more instantaneous atmospheric similarity score (i.e., how similar is this pixel to what’d we’d expect to see in a perfectly a clear reference) while cs_cdf captures an expectation of the estimated score through time (i.e., if we had all the scores for this pixel through time, how would this score rank?), and we’ve found each to have valuable advantages for different use cases. The cs band tends to be more sensitive to haze and cloud edges (which is great if you need only the absolute clearest pixels) while cs_cdf tends to be less sensitive to these low-magnitude spectral changes as well as terrain shadows (potentially giving you more “usable” pixels to work with).

Comparing masks generated with cs versus cs_cdf

• Try applying cs and cs_cdf to individual Sentinel-2 acquisitions using our Cloud Score+ Collection Explorer.
• You can also create and explore median composites using our Cloud Score+ Composite Tool.

How Cloud Score+ works

Now you may be wondering, how does the Cloud Score+ model detect clouds and cloud shadows? Answer: the power of AI! Cloud Score+ uses a weakly supervised deep learning approach to grade the quality of individual image pixels and assign per-pixel quality assessment scores.

The Cloud Score+ model was designed based on three key observations:

1. Relatively clear images from the same location look the same.
2. Determining if images are from the same place requires understanding the difference between land and atmospheric effects.
3. The atmosphere is a volume and clouds and their occlusion effects are continuous phenomena.

Which of these eight Sentinel-2 images are from the same location? How can you tell?

We begin with the observation that images from the same location tend to look very similar, while images from different locations tend to look, well, different. To exploit this knowledge, we can simply take advantage of the fact that Sentinel-2 imagery, like other operational remote sensing products, is available in a georeferenced and orthorectified format. This allows us to train an unsupervised backbone on a seemingly infinite stream of image pairs sampled from the same and from different locations that challenge the model to identify pixel positions where locations matched. Various augmentations are used to ensure the feature extractor would be robust to spectral measurement noise, misregistration, and transient landscape changes like floods, while keeping the pre-training task interesting. By requiring this feature extractor to label pixels as being from the same or different locations, the backbone “learns” to differentiate between land and atmosphere, since it must ignore atmospheric effects to determine if pixels are sampled from the sample location on the ground.

In a second stage of learning, the unsupervised “same location” feature extractor is fine-tuned to predict an Atmospheric Similarity Index Measure (ASIM). ASIM scores were calibrated using a mere 900 annotated image pairs, of which only 512 were sparsely hand-annotated. These examples represent extremely challenging cases, including very small and very thin clouds, sneaky shadows, as well as examples to ignore like phenologies, snowstorms, more floods, etc. This stage of the training process established a ranked occlusion score where atmospheric occlusion between a given pair of images was scored as Occluded, Unclear, Clear/Occluded, or Clear and assigned an alpha value representing confidence in the label. We further supplement these annotations with a continuous stream of synthesized cloud and shadow examples, which can be generated on-the-fly and provide an additional source of auto-generated “truth” for fine-tuning. Ultimately, the ASIM model produces a continuous score capable of differentiating, not between various categories of atmospheric occlusion, but rather a continuous grade of how similar the atmospheric conditions of pixels/images are.

We could stop here — but the problem is the ASIM model requires pairs of image inputs and we need an instantaneous single-image prediction for places where we might not have clear reference images. To build a QA model that can generate single-image predictions, we bootstrap ASIM predictions to a giant training dataset consisting of time series of Sentinel-2 imagery. These time series were selected by comparing pairs of images within a given time period and comparable time periods from other years and predicting the mean and standard deviation of their ASIM scores. Image-level ASIM summary statistics were then used to identify clear references, i.e., image pairs with a very high ASIM score (very similar) as well as “interesting” images. If a sampled location included at least two references and 10 supporting images that were relatively distinct/interesting, the example was selected for training. This resulted in over 2.2M training clips sampled from twelve UTM zones covering particularly challenging surface conditions.

Cloud Score+ inputs

We take these series of images, essentially short video clips, as a temporal query to train a model to generate our final Cloud Score+ QA scores. We re-use a similar feature extraction architecture to that of the initial ASIM model, but this time we run the extraction for a target image as well as images in the support set. We combine the results with an attention pool, resulting in a model that can be used to grade a target image with or without a set of support or “query” images and is able to identify surface occlusion from both clouds and shadows with high accuracy, even in challenging conditions such as over snow or ice.

For more on the model and comparisons with other datasets, check out our paper.

The final Cloud Score+ QA model is relatively tiny as neural networks go (~9.6M parameters); however, running at scale has a huge cost in terms of memory, so we optimized production so you don’t have to — we’re currently running Cloud Score+ for every new and archived Sentinel-2 image, starting with present and working our way backwards through the L1C collection.

We are excited to see how Cloud Score+ can help you to improve your Earth Engine workflows for clear compositing and beyond!

Check out these beautiful, cloud-free composites of California and the Nile River Delta, or try generating your own.

Releases & Updates

• [2024–01–16] Global backfill for the entire Sentinel-2 archive (2015–06–27 through present) is now complete and all results are available in the Cloud Score+ Image Collection!
• [2023–12–15] Global backfill through January 2018 is now complete.
• [2023–11–30] Global backfill through January 2019 is now complete.
• [2023–11–13] Global backfill through January 2020 is now complete and all 2020 imagery has been made available in the Cloud Score+ Image Collection.
• [2023–10–24] We have now completed global backfill through January 2021 and all 2021 imagery has been made available in the Cloud Score+ Image Collection.
• [2023–10–10] We announced the launch of Cloud Score+ for Sentinel-2 at Geo for Good 2023, with ongoing processing of newly acquired imagery plus global backfill through January 2022.