Mapbox challenge final round: from mapnik (raster) to mapbox studio (vector)

The #mapmadness18 is a challenge made by Mapbox so we can easily empower our maps with their tools. I’m taking this opportunity to do a proof of concept working with vector tiles because, at Homeday (the company I’m working for right now), we’re still using mapnik stack, which is super fast and reliable, to provide a huge amount of GIS data but it still is raster tiles.

Homeday’s PreisAtlas

Homeday’s new and interactive Preisatlas provides a more transparent way to check prices for renting and buying properties. Try it now and compare prices all over Germany with just a few clicks:

Mitte, Berlin - Immobilienpreise | HOMEDAY-PREISATLAS

To end up with this beautiful product and proper price estimations, everything started by acquiring millions and millions of real estate properties to be sold and rented, trying to gather as many attributes as we can for each of these properties, not only internal attributes like number of rooms, square meters, and floor, but also external aspects that influence a lot on the price like proximity to restaurants, public transportation, gyms and so on. Everyone knows that location matters a lot to estimate prices, and this is why we are heavily using OSM data too.

We’ve spent months gathering, cleaning, merging, organizing and preparing the data to feed our machine learning models. To go in details about the machine learning process, refer to this article, published by our Data Scientist, Artiom Kovnatsky. My job was to generate geo analysis and reports to be consumed by the machine earning models.

Then, we decided to go with the mapnik GIS stack to actually display the final results in terms of location:

This is working really well and fast. However, we know that vector tiles can deliver some great benefits to the user experience.

Path to convert to Mapbox stack

As a proof of concept to use Mapbox, I’ll be working only with a set of our geometries, containing blocks and buildings of Berlin with their final living quality value to be displayed on the map.

The first step to start using Mapbox Studio, and eventually Mapbox GL JS, is to convert the data from a PostGIS geo enabled table to a supported format that can be uploaded directly to the Studio. To do that:

1. With ogr2gr, export the tables from PostGIS to a geojson file;
2. With Tippecanoe, convert the geojson file into a *.mbtiles file
3. upload the *.mbtiles to Mapbox Studio

Everything gets harder when you have a large amount of data. This is why I’ve chosen to use Tippecanoe, which transforms and simplifies geo data into vector tiles tilesets, drastically reducing the storage space consumed. As we’re dealing with millions of buildings and blocks of buildings all over Germany, this is extremely important.

1. ogr2ogr: from PostGIS to geojson file

Tippecanoe will read a geojson to convert it into a .mbtiles format, which can be later uploaded directly to Mapbox Studio. I’ve used ogr2ogr to do this:

ogr2ogr -f GeoJSON blocks.json “PG:host=localhost dbname=x user=x password=x port=xxxx”

2. Tippecanoe: from geojson to mbtiles

There are plenty options to create a mbtiles file with Tippecanoe, depending on how much you want to simplify, how many zoom levels you want to cover and so on. To learn more, check out Tippecanoe repository here. In my case, I’m using these options:

tippecanoe -o blocks.mbtiles --minimum-zoom=15 --maximum-zoom=18 --drop-densest-as-needed blocks.json

3. Mapbox Studio: creating new Tilesets

With the .mbtiles file it is super easy to upload it through Mapbox Studio and create a Tileset. While logged on the Studio, inside the Tilesets tab there is a button “New tileset” which will let upload it.

Mapbox Studio setup

1. I’ve chosen the “Bright” style to start customizing my own map:

2. Then, I added my fresh new vector tiles blocks layer. If you have an uploaded Tileset available, it will be displayed here:

3. And finally, I’ve defined some rules to give color to my new layer, considering the living quality level of each block:

4. Yayyyyyy! Vector Tiles!

Mapbox GL JS: consuming the service

I’ve started with a simple service:

<script>
  mapboxgl.accessToken = "XXXX";

  var map = new mapboxgl.Map({
    container: "map",
    style: "mapbox://styles/homeday-maps/xxx",
    zoom:11,
    center: [13.397640, 52.520637]
  });
</script>

Then, I thought it would be nice to have a heatmap of some current properties available for rent in Berlin. So once again I’ve created a geojson of the properties to load them by on-the-fly and used the mapbox GL JS heatmap layer:

// ...
map.on('load', function() {
  // Add a geojson point source.
  // Heatmap layers also work with a vector tile source.
  map.addSource('properties', {
    "type": "geojson",
    "data": "props.geojson"
  });

  map.addLayer({
    "id": "properties-heat",
    "type": "heatmap",
    "source": "properties",
    "maxzoom": 14,
    "paint": {
// ...

The code above is just a sample. Here’s the complete tutorial to use the heatmap function.

And this is the final result:

You can check the project GitHub repository here.

Mapbox Vector Tiles benefits

The benefits of Vector Tiles are gigantic. They load faster, zoom levels are seemless, and it’s possible to change their style on-the-fly, which is awesome.

To be continued…

The next challenge would be converting everything into a compatible vector tiles structure. As we’re dealing with a large data set, maybe it makes sense to split everything into smaller pieces, separating the data into regions (cities?) and having many .mbtiles to be uploaded to Mapbox Studio. It’s very promising and I’m excited to continue the challenge, even after the end of the #mapmadness18.