COG Talk 4ter — Distributed processes

Last week in COG Talk 4 we discussed large scale processing using Cloud Optimized GeoTIFF and MosaicJSON. Here is another example on how we can use mosaicJSON and dynamic tiler to create high resolution large scale mosaic.

Divide, Select and Conquer

With COG and dynamic tiling you create tiles at the time of request from the raw data. Usually this lets you apply rescaling or color correction to enable a better look for web map display. With mosaicJSON and rio-tiler-mosaic, introduced in COG Talk 2, we extended the idea of dynamic tiling by adding the pixel selection operation. When we have multiple overlapping datasets, you can tell rio-tiler-mosaic which pixel you want to keep or what operation you want to perform on the stack of pixel.

Pixel selection methods applied on Landsat-8 NDVI values for all 2018 observations over Montreal area.

In this post we want to show how we can use the combination of COGs, dynamic tiler and mosaicJSON to create large scale, good looking and high resolution mosaic of Landsat 8 data.

1. Create mosaicJSON

For this demo, we are using our awspds-mosaic stack. The stack has a mosaic/create endpoint which accepts sat-api queries to create mosaicJSON of Landsat 8 data hosted on AWS PDS.

In 👆 the above we:

• define an area of interest (AOI), date range and cloud filter for the SAT API search endoint (Note: it should work the same with any other STAC api)
• POST the query to the mosaic endpoint with optional parameters (season, tile format…)

The result of the requests is a tileJSON like object.

2. Create list of mercator tiles

We are going to distribute our processes using web mercator tiles at zoom 11 (512x512 px tiles have the same resolution as zoom12 256x256 tiles). For our area of interest, this represents 783 tiles.

3. Create tile URL

In this step we define the creation option for the tiles. The query_params will be added to the tile url obtained in 1.

• bands="4,3,2": We pass the RGB band combination as a coma separated list. Here 4,3,2 correspond to Landsat 8 band combination for True Color.
• color_ops="gamma RGB 3.5, saturation 1.7, sigmoidal RGB 15 0.35": The Landsat data is stored as Uint16 data type, in order to obtain a good looking result we apply a rio-color formula.
• pixel_selection="median": This is were the magic happens. The median pixel selection option means that for each tile, the dynamic tiler will return the median value for the whole stack of data.

Note: In the _worker function you could likely add an inference step to apply ML on the output tile.

4. Distribute and collect

We now can call the tiler and get the resulting tiles (⚠ it can take up to 1 min). We also use some Rasterio code to merge the tiles into one raster file which we then translate to Cloud Optimized GeoTIFF.

5. The result

High Resolution mosaic over Britany using median pixel selection for all 2019’s spring and summer Landsat 8 scenes with less than 5% of cloud.

Doing this over Brittany is great, but can we scale it to a country size area ? 👇

High Resolution mosaic over France using median pixel selection for all summer Landsat 8 scenes (2013 → 2019) with less than 10% of cloud.

Checkout the high resolution image here.

Doing this over an entire country takes a bit more time, however using AWS Lambda and our cogeo/mosaic-* tools it takes less than 10 minutes 😱. Here are some numbers to put this in context:

• ~7 min for the tiles creation step (it takes almost longer to merge the tiles into one COG)
• 7802 number of Zoom 11 tiles to fetch (== AWS Lambda calls)
• 522 different Landsat 8 scenes (x3 bands)
• >76 Gb of data fetched (522 x 3 x ~50Mb per file)
• 48128 x 42496 output raster size

The whole notebook can be found here: https://github.com/developmentseed/awspds-mosaic/blob/master/notebooks/LargeScaleMosaic.ipynb

Got Data ?

We’re always looking for interesting problems to tackle using COGs, if you have a raster dataset and want to learn how COG, STAC or mosaicJSON could help, please feel free to ping me on Twitter or LinkedIn! And if you are interested in joining Development Seed to help us build technology that helps solve global challenges take a look at our open positions!