Looking under the hood of the Global Wind Atlas: we don’t see any typical GIS map server, but GDAL services, COG-files and 3d plotted color scales
Posted on March 27, 2019 — Ides Bauwens, Ashish Vaidya
Since the launch of version 2.0 in November 2017, the Global Wind Atlas has kept evolving towards a multi-functional and user-friendly interactive wind assessment tool. It helps wind energy specialists from around the globe to assess the wind energy potential on any given location on earth. Just recently, the atlas has faced important updates, allowing users to print and download any wind layer for any area directly from the map interface.
More than 10.000 wind energy seekers per month use the Global Wind Atlas to find wind resource hot spots anywhere on earth. Recently, wind layers have been upgraded to 250m resolution. The Global Wind Atlas is now serving 12 global layers, good for more than 160.000.000 (!) map tiles. The web interface is designed to make it very easy for users to switch between layers and regions. Visitors can visualize, print and download these wind layers for further exploration in their own environment. Downloads and prints can be customized to areas of interest, legend colors can be scaled to local min-max values. On top of that, the application can be easily extended to future new layers and is scalable to handle increasing traffic. We do all that without using any typical map server. Why? Because we believed we could do it without :).
Working with global raster layers
When working with raster layers in interactive maps, we normally generate raster tiles beforehand (pre-rendered tiles served using WMTS, TMS, XYZ) or generate them on-the-fly (WMS,…). One of the most convenient approaches for the first option is to pre-process raster tiles using the GDAL library (gdal2tiles). You simply run the command on your pc and upload all tiles to the cloud when ready. Easy Peasy.
Now have a look at the amount of tiles you would need to generate when working with global layers.
If we want to cover the globe on the highest zoom level we have to upload more than 274.000.000.000 tiles to the cloud. Let’s assume we’re using AWS S3 storage at a fixed rate of 0.005$ per 1.000 put requests. Uploading all are tiles would cost us 1.370.000$ in that case. Come again? Yeah, right.
Generating raster tiles on-the-fly using COG-files
As an alternative we coud setup a Map server (Geoserver, …) and render tiles on demand (WMS, …). But remember, we said we want to do it without using a map server right?
We can store the global Geotiff file in the cloud, but how do we start processing a file of several GB’s to generate raster tiles, convert them to .png’s and send them to the browser?
Say Hi to COG.
COG stands for Cloud Optimized Geotiff. It’s a great imagery format for cloud-native geospatial processing. GDAL allows to generate it quite easily. You can read all about it here. Said simply, COG-formated tif files allow us to request only the part of the global geotiff files we’re looking at in our map viewer. We’re using HTTP GET range requests, that allows us to ask for only the portions of the file that we need.
Now we can access only that part of the Global Geotiff file that we’re looking at in our map view. But we still need to generate tiles from it.
Say Hi to Server-Free Web Map Tile Generation.
Server-Free Web Map Tile Generation is using AWS Lambda to generate tiles from the COG-files. Chris Henrick explains very well how to get started with the AWS Lambda Tiler here.
Now we are able to generate raster tiles on-the-fly from our global COG files.
The perfect mix
In the Global Wind Atlas, we’re aiming for the perfect mix. We’re pre-processing raster tiles for the first 10 zoom levels and dynamically move to tiles-on-the-fly for higher zoom levels. This technique is the most cost-efficient approach and allows us to satisfy both data provider (World Bank Group and DTU) and the users (Wind Energy Seekers), who get to access the data for free.
Download GIS files for custom areas
Another great advantage of using COG-files is that it’s become way easier to let the user download clipped TIFF files for customized areas on the map. The Global Wind Atlas application provides users with an intuitive UI to select appropriate areas from the map. Once an area is drawn on the map, the user can with one-click simply download the original GIS data (TIFF file) from the browser.
How do we do this? We have a GDAL engine running in the cloud, which we use to clip the area of interest from the selected area. Having a microservice architecture, the requests for the file clipping are forwarded to the GDAL service that wraps the GDAL process. This also allows us to have multiple instances of GDAL service to handle bigger loads. After few seconds, the user receives a geojson file for the selected polygon, together with the clipped tiff file.
Dynamic color scaling for map printing
Last but not least, we also wanted the user to be able to export the map to a pdf or png, where the user could not only clip the map layers to their area of interest, but also rescale the color values for the clipped area.
Clipping the map
To clip the layer, we used the Boundary Canvas (https://github.com/aparshin/leaflet-boundary-canvas) plugin for leaflet, with some modifications to allow extra callbacks and cross domain calls.
Rescaling the colors
Rescaling the raster tiles colors is quite tricky since the layers that are visible to the users are already converted to image format from the original raw COG files. In other words, the raw measurement values from the layers are no longer available in the browser. They were already converted to RGB values of a tile images based on a specified legend color. At this point, we had 2 options to move forward.
- We could work with the original COG file and rescale only the required section of the layer based on the user requirement. While this seems like a straight forward approach, there are some significant down sides. The memory requirement and the long processing time of this approach means that we would have to handle the user request in batches. The requests would be processed in a queue and the users would be informed when their file is ready for download. Similar approach is applied by many companies to handle large data processing workflows but it is quite inconvenient from the user experience point of view. This approach would also not be able to provide a preview of the final result, so the user would have to wait for the processing to finish in order to check if the rescaling parameters were appropriate.
- Second option is to recover the raw values encoded in the RGB values of the tiles. To understand how this can be achieved, lets plot our legend colors in a 3d plot with Red, Green and Blue components as 3 axes x, y and z.
Since these legend points were initially used to generate the layer tiles, they can be used as the reference points for which the original values are known. Also, since the tiles are colored in a continuous color range, we can assume that not a lot of original information from the raw value has been lost during the conversion. Since we have the rgb values of the data points, we can get the closest two legend point in the above color 3d space. Now we know that the raw value of the data point falls between the values of these legend point. We can again use linear interpolation to get the precise raw values based on these three points. Ideally, the data point falls exactly on the line joining the two legend point but a lot of points do not. For this reason, we have to project the point on the line connecting the legend points. This new point is used to retrieve the original values, based on the known values from the two legend points.
Printing at high resolution
To capture the maps consistently at a higher resolution than user’s native resolution, we use headless chrome as a service to load a page that represents the map to be exported. We created a special page in the app that only has the components that are necessary for the report. When the page is loaded, we take a screenshot of the page and send the png image back to the gwa app.
Proud to be a trusted wind mapping partner
So now you had a view on how the Global Wind Atlas is working under the hood. We at nazka are quite proud to have the chance to build the interface for this fascinating endeavor, providing detailed and global wind resource data for all to access for free. It really nails our mission: developing mapps that matter.
The Global Wind Atlas is a free, web-based application developed to help policymakers and investors identify potential high-wind areas for wind power generation virtually anywhere in the world, and perform preliminary calculations. The tool facilitates online queries and provides freely downloadable datasets based on the latest input data and modeling methodologies. This version of the Global Wind Atlas (GWA 2.x) is the product of a partnership between the Department of Wind Energy at the Technical University of Denmark (DTU Wind Energy), Vortex and the World Bank Group (consisting of The World Bank and the International Finance Corporation, or IFC). Work on GWA 2.0 was primarily funded by the Energy Sector Management Assistance Program (ESMAP). The GWA interface has been reimagined and developed from the bottom up by Nazka Mapps.
