Creating a Population Density Map of Toyota City, Japan Using d3-geo’s CLI (and some other stuff) — Part 1

This is a tutorial series based on Mike Bostock’s Command-Line Cartography. Read part 2 and part 3 of this series here.

Note: What was going to be part 2 of this series has been added into part 1. Originally, part 1 went over separate population data, but I realized this is not necessary as the boundary data already has all the information necessary. If anyone was following along waiting for part 2, sorry for the change in format. If you feel stuck have a look at part 2 of Mike Bostock’s tutorial, which I’m basing this on.

This post will document how I go about making a choropleth map of Japan showing population density using d3. I will actually be basing this off of Mike Bostock’s Command-Line Cartography tutorial series. So it might be useful if you are confused about anything in that tutorial, or if you are interested in using open data from the Japanese government, which is where we’ll get our data.

Quick note, you may have never heard of Toyota City. It’s a relatively small city on the outskirts of Nagoya (a large major city) with a bit less than half a million people. I lived there for a year and really enjoyed it, that is the only reason I’m choosing this city.

Let’s get right into it. First, we’ll need to download the data to work with. For this project, we’ll only need one set of data: Cartographic boundary data. This is the data for the actual boundary data of Toyota City, including all the smaller administrative boundaries within the city. In this case, we’ll be using official government administrative boundary data from the 2015 Japanese Census. The great thing about using official boundary data like this is that there is official census data which corresponds to each individual area. For this map that will allow us to see the population of each individual area.

NOTE: Japanese boundary data includes area and population! This is everything we need to compute population density. If you are using data from another source or country, you may need a separate dataset which contains population data.

Boundary Data

To download the cartographic boundary data, go to the Japanese government’s open data portal, e-Stat. That link will take you to a page with different administrative areas. We want to click on the blue box to the right of the row that says “23211 豊田市”. This will get us a shapefile version of our Toyota City administrative boundary data. Shapefile is a spatial data format created by ESRI, the makers of ArcGIS. It’s probably the most widely used format so it can function with many tools.

Let’s check to see if our data is what we think it is. If you have a GIS suite (QGIS or ESRI) you can throw it in there and see what comes up. But for this tutorial, we’ll use a really handy web tool called Mapshaper. First off, let’s go to the Mapshaper website. Drag and drop the .shp file you just downloaded (there will be others as well, but just use the .shp). Leave all the settings as they are for the dialog box that comes up and click “import”. If you see something similar to the picture below, you’re good to go!

The administrative boundaries of beautiful Toyota City!

So we’ve confirmed that we have working shapefile data. Next, we’ll get the population data which corresponds to each of these administrative areas. While shapefile data is handy for many things, there are forms which are better suited for the web and for D3. A popular format is GeoJSON, a version of the popular JSON data format. The great thing about JSON and GeoJSON is that they are written in Javascript, and thus are easy to implement on the web.

Editing Boundary Data

In Mike Bostock’s tutorial, he uses his npm tool called “shapefile” in order to convert shapefile data to GeoJSON data. This will probably work in most situations, but it wasn’t handling the Japanese characters which are in our data, and output them as strange, random looking characters. I even tried encoding the shapefile as UTF-8, still no luck. Luckily, I found the our trusty Mapshaper website we used before does the trick. Just put in the shapefile data (drag and drop all of the shapefiles this time) in, and there will be an option in the top right corner to export. Click this and export as GeoJSON. Open it up is an editor and it should look something like this:

As you can see, our Japanese characters are a-ok here.

The next step is to properly project our data. When I say project I mean choose a projection such as the Mercator or Albers, which you may have heard of. What you not have heard is that there are tons of these projections which are used for different regions of the world and for different purposes. We want to project our data now because it is faster to display pre-projected data rather than using computing power to project it properly upon display in a browser. There are tons of ways to project your data, and I’ll be honest, I had some issues actually getting it properly projected for this project. The way I’ll show here is the same way Mike Bostock shows in his tutorial, using terminal commands for d3. This way is honestly very painful compared to others, but it does allow us to fit our map into a bounding box for when we export it as an svg image. So let’s install the proper d3 tool:

npm install -g d3-geo-projection

Now, the projection we’ll be using is EPSG:3099. This is one specifically for displaying medium scale topography in the area of Japan Toyota falls in. If you are looking for a specific projection for a different geographic area, the Spatial Reference website is a good resource. Unfortunately, there is no option in d3 to simply input the projection identifier into, instead, we need to dig into its specifications and extract the necessary info. Like I said, a bit painful but that’s what we’ll do. This is what the command ends up as:

geoproject 'd3.geoTransverseMercator().rotate([-135, 0]).fitSize([960, 960], d)' < toyota-data.json > toyota-proj.json

A little explanation of the above: ‘geoproject’ is one of the tools we just installed. In the EPSG:3099 specifications, it states the projection it uses is ‘transverse mercator’, so we put that in as the d3 projection. We take the ‘central meridian’ and ‘latitude of origin’ values and put them in d3’s ‘rotate()’ function, with the central meridian being negative. The ‘fitSize()’ function is what is most useful about geoproject here, it fits our data into our 960 by 960 bounding box.

Great, now we have a properly projected GeoJSON file of Toyota! Let’s have a look at it by creating an svg image of it using another d3 cli command:

geo2svg -w 960 -h 960 < toyota-proj.json > toyota-proj.svg

Quite different from the unprojected form in Mapshaper

The next step is to convert this data into a format that will make it easy to work with and edit in the command line. To do this we’ll use ndjson-cli, a command-line module for converting JSON files to newline-delimited JSON files (NDJSON). Basically JSON formatted so that each line is a value. In this case, each line of our GeoJSON file will be a feature (feature is GeoJSON terminology). To get started, let’s install ndjson-cli:

npm install -g ndjson-cli

Now to create an NDJSON formatted file from our data, splitting it into one feature per line:

ndjson-split 'd.features' < toyota-fit.json > toyota.ndjson

Not only is this handy for manipulating data, it’s much more readable as well:

Only the first 2 lines are visible here, but they each contain all the info for one feature

The properties on interest to us are AREA, which contains the area of the feature in square meters, and JINKO, which contains the population of the feature. Because they are both conveniently already attached to the data, we can calculate the density. The command below will use ndjson-map to do that while dropping all the other properties we don’t need, which will make our file smaller. For more information about ndjson-map, see the documentation:

ndjson-map 'd.properties = {density: Math.floor(d.properties.JINKO / (d.properties.AREA / 10000))}, d' < toyota.ndjson > toyota-density.ndjson

AREA is divided by 1000 to convert the measurement to people per hectare (10000 square meters). The result should look like this:

density is now the only property

Now let’s transform our data back into regular old GeoJSON using ndjson-reduce and ndjson-map:

ndjson-reduce \
  < toyota-density.ndjson \
  | ndjson-map '{type: "FeatureCollection", features: d}' \
  > toyota-density.json

If we throw it into Mapshaper:

We get an upside down map. This is because Mapshaper and SVG treat the y axis opposite. So it will appear normally in SVG.

Now let’s use D3 to make a quick choropleth in the terminal. First we install D3:

npm install -g d3

Now we’ll use ndjson-map with D3 (via ‘-r d3') to create a ‘fill’ property using the Viridis color scheme. We’ll be using our previous NDJSON file here:

ndjson-map -r d3 \
  '(d.properties.fill = 
d3.scaleSequential(d3.interpolateViridis).domain([0, 265])
(d.properties.density), d)' \
  < toyota-density.ndjson \
  > toyota-color.ndjson

I have put 265 people per hectare as the max value here. I found out the max value by inputting this command:

ndjson-top 'a.properties.density - b.properties.density' \
  < toyota-density.ndjson

This will spit the whole file in your terminal, with the last feature being having the highest density value.

Now convert it to SVG with geo2svg:

geo2svg -n --stroke none -p 1 -w 960 -h 960 \
  < toyota-color.ndjson \
  > toyota-color.svg 

Right-side up this time!

Ok so this looks quite pretty and if you compare it to an aerial image of Toyota it seems to make sense. We still need to make it so we can more easily distinguish between population density in the lower density area though, as well as add important things like a key. But we’ll get there in future parts!

Part 2