How to create your own Cloudless Mosaic in less than an hour
Perform preparatory analysis and use Sentinel Hub Batch Processing API within Requests Builder to ease your work!
Only approximately 30% of skies over land are typically cloud-free at any given point in time, complicating synoptic monitoring of the Earth’s surface using optical satellite imagery. A well-known solution to circumvent this problem is to create cloudless mosaics, which consist of multiple observations acquired over a given time period that are cleverly stitched together to form a single image. However, creating cloudless mosaics over large areas requires substantial bandwidth to download large volumes and significant processing power to stitch the images (sometimes dozens of computation hours).
In 2018, Sinergise wrote an article on How to Create a Cloudless Mosaic using EO Browser’s temporal abilities, saving users the hassle of downloading and processing the images, as the grunt work was done by Sentinel Hub services. With our support, Pierre Markuse even produced a 64.000 x 64.000 pixel mosaic of Europe!
Earlier this year, Grega Milcinski introduced Sentinel Hub’s Batch Processing API in this blog post, showcasing how we deal with satellite data requests over large areas and/or long time periods. In this feature, we are going to generate a cloudless mosaic of a notoriously cloudy area of the world. The Batch Processing API perfectly fits the purpose because we need to process months of satellite data over a large area to generate the cloudless mosaic. The API takes over most of the computationally intensive steps in the workflow, speeds them up through parallel processing and provides us with the processed data.
So now grab your laptop, a cup of coffee and get prepared to generate cloudless mosaics in less than an hour all within the comfort of your favourite web browser!
Cloud Statistics
Before jumping into creating your own cloudless mosaics though, let‘s get you equipped with a useful tool allowing you to check if there are enough sufficiently cloud-free images in your area of interest over the selected time period. Based on our s2cloudless algorithm and the Sen2Cor scene classification (SCL) data, the Cloud Statistics custom script counts the number of cloud-free Sentinel-2 L2A pixels for a given time range. Then it visualises the ratio of these compared to the total number of pixels in the time period. In the figure below, you can see the effect of selecting different time-ranges (5 months vs 1 month of data) on the cloud-free pixel ratio over Newfoundland (which hereinafter will serve as our area of interest). If you were to only request images from June 2019 for processing a cloudless mosaic of the area, this would most likely leave you with cloud artefacts in your mosaic since black pixels mean there are no existing cloud-free pixels for the given location and time range.
Every journey begins with some preparations, but the view at the end is going to reward you for that comparatively small effort you are putting now. This simple hack is going to help you pave the way to a perfect cloudless mosaic, as you can get a better idea about the data availability before you start your Batch Processing.
The Requests Builder
The Requests Builder started as an auxiliary tool for internal use with the goal to simplify the creation of Process API requests and became a powerful application (currently in beta version) that enables one to access all functionalities of our Process, Batch, Third-Party Data Import APIs and OGC services through a user-friendly interface directly in your browser, without having to stress about correct JSON or javascript syntax. Following the logical workflow of API request creation, the interface is divided into different main panels in which you need to select the usual in- and output parameters. The tool lets you choose an area of interest (AOI) on an interactive map and even offers a javascript syntax validation (linter) for the Evalscript.
Note: You can deploy the Requests Builder as an app in the Euro Data Cube!
Tailor your Batch Processing Request
Simply select the BATCH radio button on the top of the screen after logging in with your Sentinel Hub user credentials. Then, tailor parameters such as datasource, time range and response output to your liking and choose your area of interest in the interactive map. You can draw a bounding box or a polygon, paste available geometry coordinates to be parsed by the tool, or upload a KML/GeoJSON file. Don’t forget to specify the output parameters for the low-res preview, which triggers a Process API request that returns a low-resolution image of the selected area. A visual interpretation of the preview image allows you to validate the applied request parameters and also check the correctness of your Evalscript before running a faulty batch request that is not returning the desired results.
Evalscript for Cloudless Mosaics
The second main panel allows the input of an Evalscript that suits your needs. Directly edit it in the related tab or simply copy and paste it. Whatever changes you make to the Evalscript or request parameters, the Requests Builder will automatically account for in real-time and adjust the Request Preview on the right side.
For our Newfoundland example, we enhanced this script for the creation of cloudless mosaics from the custom scripts repository as follows:
- We implemented s2cloudless cloud mask data by adding
CLMto the bands in thesetupfunction (line 6 in the enhanced script version below) and to thevalidatefunction (line 36 and following), which groups pixels from all available timestamps into valid and invalid ones based on the cloud mask overlay and no-data values. - We adjusted the
filterScenesfunction to only include data up to 5 months (line 19) before the latest available scene in the specified time range. This acts as an additional safeguard so you are not requesting satellite imagery from 1st June 2017 until 31st October 2019 because you accidentally selected the wrong year.
Don’t know how to get started with your Evalscript? Get lots of prefabricated scripts for a large variety of use-cases from our custom scripts repository and adapt them to your needs. Make sure to read this blog post and feel free to share your own awesome scripts with the community!
Test the enhanced cloudless mosaics script in our EO Browser and adapt it to your needs! For example, try out a false-colour infrared representation for large area vegetation analysis.
Low-Res Preview
After having adapted the Evalscript to your request it is time to test if the Evalscript is returning the expected results and if we can make out any problematic areas (e.g. cloud artefacts) in the low-resolution preview. The preview can be requested by clicking on the Get Low Res Preview button in the third main panel. Depending on the size of the area and the dimensions selected for the preview, this process might take some seconds or minutes.
The Batch Processing Request
Now that you have validated your request with the preview, it is time to create the actual Batch Processing request. Have a quick look at the available commands and response statuses for batch requests in the figure below.
Select a tiling grid and resolution that fits your needs and provide an S3 bucket name for storing the resulting image tiles (more info here). The Create button generates your Batch Request with a unique Id. You can always check the current status of your request and additional information, like an estimated number of output tiles to process, in the Batch Information tab on the right.
To get a cost estimation of the request value in Processing Units, you should analyse your created request by clicking the Analyse button. The request status first changes to ANALYSING and to ANALYSIS_DONE when it is finished. Again, you can see the latest status and additional information like the Estimated Value (in Processing Units) of the request in processing units in the Batch Information tab.
Note: Batch Processing requests cost a third of a normal Process API request!
Now you can start processing the tiles from your area of interest with the Start button. The progress of your request can be checked by clicking Refresh Tiles in the Batch Information tab of the respective request. Failed tiles can be restarted via the Restart Partial Request button and will be rescheduled for processing. Once all the tiles have been processed and uploaded to your s3 bucket, the status of the request changes to DONE.
Now that you have all the processed tiles sitting in your S3 bucket, how do you merge them into a single image?
Tile Download and Stitching
Of course, you can download all tiles to your local drive to merge them, but why not conveniently streaming the tiles from your AWS s3 bucket without having to store them on your local machine?? 💡
We created a little helper script that uses GDAL’s virtual file system vsis3 for on-the-fly reading of the processed tiles in your s3 bucket. The script merges all tiles into one single mosaic and can handle tiles from different UTM zones (important for larger areas that span across UTM zone extents). You need to configure the AWS Command Line Interface (CLI) with your AWS credentials as the script is calling CLI commands. Follow this link to read about the quick configuration and other configuration basics if needed.
Display the help description for the tool and necessary inputs by adding -h to the script call in your command line: ./batch-merge-tiles-vsis3.sh -h. To start the processing, you need to provide the tool with the s3 bucket name (-b), the EPSG code (-c) of the desired output coordinate reference system, the preferred interpolation method (-i) and the output path (-o). If you are using a named profile for AWS services, you can add the profile name via the optional -p flag. Call the script from the command line like this:
./batch-merge-tiles-vsis3.sh -b <s3 bucket name> -c <output CRS> -i <interpolation method> -o <output path> (optional: -p <aws profile>)
After processing, your output folder will contain the mosaicked GeoTIFF of your area of interest with two additional downscaled versions.
Take a look at the marvellous results!
Compared to the hardships you previously had to go through to create cloudless mosaics, the Batch Processing API facilitates the processing of large areas and long timespans so you can create correctly projected mosaics of your area of interest in no time! The whole island of Newfoundland spans over an area of 108,860 square kilometres and is regularly cloud-covered, so we used 5 months of Sentinel-2 L2A data to generate this 10 m resolution cloudless GeoTIFF mosaic.
Note that you can even use our “Bring your own COG” option to automatically ingest Batch results into Sentinel Hub so that they can be distributed via standard WMS APIs.
Since Batch Processing can be quite demanding in terms of processing, please get in touch with us if you want to give it a try. This feature is currently available automatically to users with an enterprise account. Let us know if you are using another plan and would like to use our Batch processing API.
- Follow our Twitter and LinkedIn for news, so you don’t miss the latest developments and features!
- Feel free to share your cloudless mosaics and Batch Processing results with us and the Sentinel Hub community via our social media channels!
- If you have questions or encounter any problems, please engage with us and other users from the community in our Sentinel Hub Forum.
