Time-lapse Scatter Map-visualization using Cartopy

Introduction (what we’ll create):

Cartopy is your choice of library if you want to create static, non-interactive images/ videos of scatter map-visualizations using a custom background. The following video, which we’ll create in this tutorial, is a good example of what is possible with Cartopy:

Structure of the tutorial:

The tutorial is structured into the following sections:

1. Pre-requisites
2. Installing Cartopy
3. Getting started with the tutorial
   
   - Importing relevant packages
   
   - Loading the data and making sense of it
   
   - Setting custom background
   
   - Generating the plots
   
   - Stitching the images to form a video
4. When to use this library
5. References

Pre-requisites:

This tutorial assumes that you are familiar with python and that you have python downloaded and installed in your machine. If you are not familiar with python but have some experience of programming in some other languages, you may still be able to follow this post, depending on your proficiency.

Installing Cartopy:

If you are using Anaconda,

conda install -c conda-forge cartopy

Otherwise, the installation process is a bit lengthy. First, you need to install the following dependencies: GEOS, NumPy, Cython, Shapely, pyshp, and six. These can be installed using the pip installer or other package managers such as apt-get (Linux) and brew (macOS). Once these dependencies are installed, Cartopy can be installed using

pip install cartopy

For more information on installing Cartopy, see https://scitools.org.uk/cartopy/docs/latest/installing.html

Getting started with the tutorial

Git repository: https://github.com/carnot-technologies/MapVisualizations

Relevant notebook: CartopyDemo.ipynb

View notebook on NBViewer: Click Here

Importing relevant packages:

import numpy as np
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy as cartopy
%matplotlib inline
import os as os

Loading the data and making sense of it:

The data set that we’ll be using is provided in the ‘data’ folder. Load the scatter_dummy_data.csv from that folder.

#Load the data
df = pd.read_csv('data\\scatter_dummy_data.csv')

On observing the contents of the dataframe, you’ll notice that it consists of the following columns:

1. day (the day on which the trip happened)
2. device_fk_id (the ID of your IoT device)
3. n_trip_on (Number of trip ON points)
4. start_lat (Latitude at the start of the trip)
5. start_lon (Longitude at the start of the trip)

One way of understanding this data is to use a ride-hailing service analogy. Suppose you have a ride-hailing service, like Uber, and you install an IoT device in each cab. This data represents the starting location for each trip and the number of points in each trip (a proxy for the duration of the trip), sent by each device every day.

We will attempt to see how the trip data changes in March, in the form of a time-lapse video. First, we’ll need to ensure that the type of the day column is datetime. For that, we explicitly convert it to datetime.

#Convert the day column to datetime
df['day'] = pd.to_datetime(df['day'],format="%d-%m-%Y")

Setting custom background:

Now, before we begin plotting the points, we want to set a custom background. The ability to set a custom world-map image as a background is the USP of this library. By default, cartopy comes with just one default background, shown below. On a Windows machine, it can be found at the following address: C:\ProgramData\Anaconda3\Lib\site-packages\cartopy\data\raster\natural_earth

The default world map that comes along with the Cartopy library

However, this background doesn’t look very aesthetic. Instead, we will use a Blue Marble map from NASA. A lot of beautiful world maps can be found here: https://visibleearth.nasa.gov/collection/1484/blue-marble.

The map that we’ll be using is shown below:

Blue Marble NASA map that we’ll be using in this tutorial

Now, in order to use this map, we need to tell Cartopy where to find it. That is accomplished by setting the ‘CARTOPY_USER_BACKGROUNDS’ environment variable.

#Set the directory for the background image
current_directory = os.getcwd()
os.environ["CARTOPY_USER_BACKGROUNDS"] = os.path.join(current_directory,'data','CARTOPY_IMGS')

If you check in the ‘data’ folder in your cloned repository, you will see a folder called ‘CARTOPY_IMGS’ containing the above Blue Marble image and an images.json file. The above piece of code assigns the path to this folder to the ‘CARTOPY_USER_BACKGROUNDS’ environment variable. In order to verify that the path has been set properly, you can query the path.

#Verify that the environment variable has been correctly set
os.getenv('CARTOPY_USER_BACKGROUNDS')

The images.json file contains the metadata of the images. If an image is physically present in the folder but not described in images.json, then Cartopy will throw up an error.

Our images.json file looks like the following:

{“__comment__”: “JSON file specifying the image to use for a given type/name and resolution. Read in by cartopy.mpl.geoaxes.read_user_background_images.”,
 “BM_NASA”: {
 “__projection__”: “PlateCarree”,
 “__comment__”: “CustomBG”,
 “__source__”: “CustomSource”,
 “low”: “BM_NASA.png” }
}

BM_NASA is the name of the image. The value for the key ‘low’ indicates which image file to use if the user has asked for the low-resolution version. We could similarly have separate files for medium and high resolution for the same image. You can find out more about the background image in cartopy here.

Generating the plots:

Now that we have the background map, we’ll first limit its extent only to India.

#Define the extent of the plot. 
#Here the extreme lat lon of India are hard-coded.
lat_min = 8.06890
lat_max = 37.08586
lon_min = 68.1941
lon_max = 97.39564
extent = [lon_min, lon_max, lat_min, lat_max]
central_lon = np.mean(extent[:2])
central_lat = np.mean(extent[2:])

With both the background and the image set, we can now generate images for each day. The following snippet will generate 29 images, for data belonging to 1st to 29th of March, in the frames_cartopy folder:

fig = plt.figure(figsize=(24, 12))
ax = plt.axes(projection=ccrs.PlateCarree(central_lon))
for i in range(1,30):
    date = datetime(2020,3,i)
    df_single_day = df[df['day'] == date]
    if ax is None:
        fig = plt.figure(figsize=(24, 12))
        ax = plt.axes(projection=ccrs.PlateCarree(central_lon))
    ax.set_extent(extent, crs=ccrs.PlateCarree())
    #Setting the background image and creating the plot
    ax.background_img(name='BM_NASA', 
                      resolution='low',
                      extent = extent)
    
    ax.scatter(df_single_day['start_lon'], 
               df_single_day['start_lat'], 
               s=df_single_day['n_trip_on'].apply(lambda x: x/100),
               alpha=0.8, color='#FFFF50',
               transform=ccrs.PlateCarree())
    
    #Adding text
    fontsize = 36
    # Positions for the date
    date_x = 85
    date_y = 10
    ax.text(date_x, date_y, 
            f"{date.strftime('%b %d, %Y')}", 
            color='white',
            fontsize=fontsize,
            transform=ccrs.PlateCarree())
    #saving the figure. Name the files as frame_01, frame_02, etc.
    fig.savefig(f"frames_cartopy/frame_{i:02d}.png", 
                dpi=100,facecolor='black', bbox_inches='tight')
    
    #Clear all the figures except the last one. 
    #We'll show the last one in-line in the notebook.
    if(i != 29):
        ax.clear()

While the comments in the above snippet make it self-explanatory, I’ll explain a couple of important lines.

#Setting the background image and creating the plot
    ax.background_img(name='BM_NASA', 
                      resolution='low',
                      extent = extent)
    
    ax.scatter(df_single_day['start_lon'], 
               df_single_day['start_lat'], 
               s=df_single_day['n_trip_on'].apply(lambda x: x/100),
               alpha=0.8, color='#FFFF50',
               transform=ccrs.PlateCarree())

Here, the ax.background_img reads the images.json file to determine which is the low-resolution version of the BM_NASA image. In the scatter plot, a scaled version of the ‘n_trip_on’ field has been used for setting the size of each individual scatter point. Thus, devices that did a longer trip will be represented by a larger circle.

The next important field is the ‘transform’ field. PlateCarree is just one of the transforms possible with Cartopy. There are several others, like Orthographic, Mercator, etc. To get the list of all transforms, along with their interpretations, click here.

Congratulations! You have created a separate visualization for each day! Now the last step is to stitch these images together to form a time-lapse video.

Stitching the images to form a video:

Now that we have the images prepared, we can now stitch them together to form our time-lapse video. For that open your terminal or command prompt, navigate to the frames_cartopy folder and run the following command:

ffmpeg -framerate 3 -i frame_%2d.png -c:v h264 -r 30 -s 1920x1080 ./cartopy_video.mp4

Here, we are using a frame rate of 3 frames per second. Our frames are named as frame_01.png, frame_02.png, up to frame_29.png. frame_%2d.png tells the command to look for files with two digits after ‘frame_’ and arrange them in the ascending order. The resolution is specified as 1920x1080 and the name of the video is cartopy_video.mp4.

To know more about the different options related to video-encoding using ffmpeg, you can visit https://trac.ffmpeg.org/wiki/Slideshow

When to use this library:

This library comes in handy when you want to create an image/ video with a custom background. It can be used for static images/videos that you want to include in presentations or websites. However, there is absolutely no interactivity that you can get with this library. If you want to host a visualization wherein people can zoom into the map/ click on the points, this is not the choice of library for you. You would want to use a library like plotly for that purpose.

References:

1. Official Cartopy documentation: https://scitools.org.uk/cartopy/docs/v0.15/index.html
2. Another example of creating a time-lapse video using Cartopy: https://medium.com/udacity/creating-map-animations-with-python-97e24040f17b

We are trying to fix some broken benches in the Indian agriculture ecosystem through technology, to improve farmers’ income. If you share the same passion join us in the pursuit, or simply drop us a line on report@carnot.co.in