Creating vector tiles with Python

Vector tiles are an amazing way to render vector data over the web. It is highly performant and flexible. The only problem is that it can be rather hard to setup. It’s not easy to cut up your vectors into tiles, simplify/aggregate them the right way and define correct zoom levels for them.

Many tools to create vector tiles have for this reason default settings, but the default might not always fit your needs so:

In this tutorial I will show a concrete example how to create vector tiles using Python.

Constructing them yourself from scratch gives you full freedom to define your rendering.

What are vector tiles?

If you are not familiar with vector tiles, here is a (very) quick intro. The idea is to carve the world up in tiles of different zoom levels. We start with 1 tile covering the flat world on zoom 1. Then we continue splitting this 1 tile into 4 tiles on zoom 2, then 16 in zoom 3 etc. Once we have enough detail for our use case we stop.

The basic principle of vector tiles

The idea is now to place vectors in each of these tiles. When looking at a certain spot on the map the client will retrieve the appropriate tiles and render the vectors. The more the user is zoomed in the more fine grained the tiles will be.

Vector tiles have 4 key advantages

1 You can define levels of detail for each zoom. So no need to render all your data all at once

2 Search efficiency. Looking for a tile is log(N) efficient, way faster then if you would search by for example bounding box.

3 Ease of rendering. Since the vector tiles are localized in a single tile clients will have an easier time rendering the vectors.

4 Caching. Since you are using predefined tiles you can easily cache them.

What are the Ellipsis Drive defaults for vector tiles?

I will be hosting my vector layer on Ellipsis Drive in this tutorial. When adding vectors to my layer, Ellipsis Drive will in the background place these vectors within tiles. By default each vector is added to each zoom from 0 to 21. The vector is added to the tile in which its middle point resides.

It will also create levels of details of each vector. So if I retrieve a certain tile I can choose the maximum number of vectors that I wish to retrieve AND the level of detail on which I would like to retrieve them.

When there are more vectors within a tile than you requested you will be returned a random sample.

When zoomed out I get a sample of the vectors in low level of detail

When I zoomed in I get all vectors in high level of detail

Since the vector is only tied to the tile in which its middle point resides, the max zoom level is chosen in such a way that the tiles are large compared to the vectors. This makes sure that vectors do not disappear if they are only partly within your screen.

These defaults make sure I can use the advantages of vector tiles without thinking or even understanding the concept. However in many cases you can improve on this default behaviour a lot by defining you own vector tiles!

So let’s do just that

I have a small data set containing country borders. (I hope the example is not too politically sensitive these days ;p ). Let’s first load the data in Python and plot it.

import geopandas as gpd

file = '/home/daniel/Downloads/6f49bb77-a83c-4fce-9b3d-072238655f8d'
features = gpd.read_file(file)
features.plot()

A quick plot of our data set

The default and why it is sub optimal

As a first attempt we can place these vectors in an Ellipsis Drive layer without thinking.

import ellipsis as el
token = el.account.logIn('MY_USERNAME', 'MY_PASSWORD')
#creating a vector layer and a timestamp withing that layer
pathId = el.path.add(pathType = 'vector', name='Medium example', token = token)['id']
timestampId = el.path.vector.timestamp.add(pathId = pathId, token = token)['id']
#placing my geopandas within the layer
el.path.vector.timestamp.feature.add(features= features, pathId = pathId, timestampId = timestampId, token = token)

When opening the layer in Ellipsis Drive I now see:

A nice map, but it takes too long to load

Not bad, but you will notice that the level of detail is not great:

The coast line of Norway lost all of its detail!

I can fix this by using the tune in the side pane. I can ask the UI to render me high level of detail polygons instead:

I can place the level of detail to high

This now fixes the detail problem. However the loading of the overview now takes a long time. Over 5 seconds!

As a last attempt I can tell the layer to render the level of detail depending on the zoom. To this end I place the max zoom on 6 and make the level of detail zoom dependent.

This fixes my issue. The overview renders quickly and when I zoom in I get better level of detail.

However I now have a new issue… When zooming in too far certain polygons disappear!

When zooming in we can still see the border of Cuba, but the border of the USA disappeared

This is because I tempered with the default max zoom. The max zoom is now causing me to use tiles that are smaller than the polygons themselves. Meaning that the middle point of the polygons do not need to be in a tile that I have in my screen.

So whatever I do, I either need to accept a long loading time, lower level of detail or occasional disappearing vectors when zooming in. If I do not want to make any concessions I will need to define my own vector tiles!

Notice that the problem occurs because the level of detail of the borders of the polygons is high when compared to the size of the polygons.

Defining my vector tiles

Creating vector tiles consist of 2 steps. In step 1 we define levels of detail. In step 2 we cut the vectors into tiles of each zoom level.

Step 1

We can create levels of detail by using the ellipsis.util.simplifyGeometries function. This function is simply the shapely.simplify function only now it makes sure to preserve geometry type and remove islands.

lod1 =  el.util.simplifyGeometries(features = features, tolerance = 1, removeIslands = True)
lod2 = el.util.simplifyGeometries(features = features, tolerance = 0.1, removeIslands = True)
lod3 = el.util.simplifyGeometries(features = features, tolerance = 0.01, removeIslands = True)
lod4 = el.util.simplifyGeometries(features = features, tolerance = 0.001, removeIslands = True)
original = features

Now we need to map each zoom level that we are going to use to a level of detail. I will use the zoom levels 0 to 6.

zoomMapping = {0:lod1, 1:lod1, 2:lod1, 3:lod2, 4:lod3, 5:lod4, 6:original}

Step 2

To cut my vectors into tiles I use the util.cutIntoTiles function from the ellipsis package that does just this for us. We can provide this function our geopandas dataframe and a desired zoomlevel. The function will return a geopandas dataframe in which geometries are cut to tiles.

For example we can cut our original features for zoom = 6

features_cut = el.util.cutIntoTiles(features = features, zoom = 6)
features_cut.plot()

This gives us the following result

Countries split over tiles at zoom=6

As you can see we still have our original border data set, but we split all the vectors into smaller tiles, so that we can render only one small tile when the user zooms in somewhere instead of the full country. This is a big win. Imagine you zoom in on a part of Russia. You can now load a lot of detail for that small regions by retrieving the correct tiles, instead of needing to download the whole contour (which contains information of points that are half a world away).

Putting it all together

So what we will now do is the following. We already have the different levels of details of the borders. But we will now loop over the zoom levels 0 to 6 and each time cut the appropriate level of detail into tiles of the zoomlevel. Once that is done we will add it to the map.

for zoom in range(6):
    features_zoom = zoomMapping[zoom]
    features_cut = el.util.cutIntoTiles(features = features_zoom, zoom = zoom)
    el.path.vector.timestamp.feature.add(zoomLevels = [zoom], features= features_cut, pathId = pathId, timestampId = timestampId, token = token)

We now have a map in which the level of detail increases as we zoom in, and no vectors ever disappear.

Quick rendering low level of detail when zoomed out

When zoomed in both high level of detail and no disappearing vectors!

If I display the lines of the vectors you can clearly see why this works. The country borders have actually been sliced to tiles!

The borders are sliced into tiles that fit the zoom tiles

You can find the interactive map of the result here.

Conclusion

By default Ellipsis Drive does a good job. It calculates a zoom so that the vectors compared to the tiles of that zoom are small and it places all vectors in a tile based on their middle point. However there are 2 cases in which you can improve on the default.

The first case is for large polygons with detailed boundaries. This is the case that we have discussed in this article. Country borders have a lot of detail compared to their respective size. In this case you are best off defining the tiles yourself.

The other case is when you have many small vectors for which a simple down sample does not suffice. By default Ellipsis Drive just down samples the number of vectors on lower zoom levels. But this is not always the best option. For example it can be preferred to collect multiple vectors together into one (think of grouping provinces into countries on low zoom) or you might want to display the most important vectors and not just a random sample (think of roads, in which you would like to display the highways first and the secondary roads only on higher zoom.

This last case we did not discuss here, but I will make another post in the future on how to create vector tiles for a road network.

PS: In case you have a very large number of vectors. Consider dumping them to a file and then using ellipsis.path.vector.timestamp.upload.add() to upload the file. This prevents you from pumping millions of vectors via API calls.