Data Mining OpenStreetMap

OpenStreetMap (OSM) is built and maintained collaboratively by a group of enthusiastic mappers. It all started in 2004 — Steve Coast was frustrated with the state of maps in the UK. Although the government had large chunks of data stashed away, the data really didn’t solve any problem. Also, the map space was closed; there were only a few vendors offering data and services and if you wanted a solution related to geospatial entities (terrain information, driving direction, city map), you would pay for data which by the way was closed for extensibility and customization. Almost around the same time, the success of Wikipedia caught Steves interest. He decided to replicate the same crowdsourced model for maps. Later that year, he founded OpenStreetMap. It is fascinating on how Steve transformed the closed mapping industry into a vibrant user community with one goal; to offer free editable map — Read more on his journey here.

In the previous posts, we talked about geoJSON maps, raster maps, vector maps and rendering maps with JS libraries. If you have been following along diligently building solutions with various tools, pat yourself on your back. You now have the power to create immersive and meaningful map solutions. Data is king. Right from the days of Neanderthals to the modern age man trying to locate the nearest Starbucks, data has made the difference between life and death. Well, finding Starbucks is not an appropriate comparison here. But you get my point; data is king. Thanks to Steve’s efforts — we now have OSM.

There are various ways (R, Java, JavaScript, Python, C++ libraries) we can surface and use OSM data. For the remainder of this post, we will use overpass-turbo, the web-based data mining tool to mine desired datasets. As always we learn by building, so let’s build something. I have always been intrigued by the number of prisons in the united states and its correlation with crime rates. How about we use Mapbox GL JS to render a map and overlay prison information retrieved from overpass-turbo? That sounds interesting! But how do we assess correlation with crime rates? Perhaps we can look at the published crime data and identify a pattern? Great. Let’s begin.

You can follow along by cloning your copy from this repo.

To get a better handle using overpass-turbo, we need to understand OpenStreetMap’s data model.

• A node indicates a point of interest.
• A way connects to nodes
• A relation represents a collection of nodes and ways
• Tags are metadata describing a node, way or a relation.

The OSM community tags each feature with a relevant description through tags which effectively enables analysis and querying. Take a look at the collection of available map features. In this lot, our interest lies with amenity=”prison” in “usa”

Pro Tip: If the query timesout, increase timeout in the query and click ‘Run’

When a query is run, the returned data is overlaid on overpass-turbo’s map. That’s pretty neat! But we want to download the dataset so that we can overlay it on our map. Lucky for us, overpass turbo knows our requirement. Click on Export -> download/copy as GeoJSON.

Moving on to overlaying the dataset on a map. Let’s render a map first.

let map = new mapboxgl.Map({
    container: 'map',
    style: 'mapbox://styles/mapbox/light-v9',
    center: [-97.38, 42.87774],
    zoom: 3
});

With Mapbox GL JS, multiple layers can be added to supplement the base map. Let’s load the prison data and add it as a layer to the base map.

map.on('load', function () {
    let xhr = new XMLHttpRequest();
    xhr.open('GET', "src/data/us_prisons.geojson");
    xhr.setRequestHeader('Content-Type', 'application/json');
    xhr.onload = function () {
        if (xhr.status === 200) {
            let data = JSON.parse(xhr.responseText);

            map.addLayer({
                "id": "prisons",
                "type": "symbol",
                "source": {
                    "type": "geojson",
                    "data": data
                },
                "layout": {
                    "icon-image": "prison-15",
                    "text-field": "{name}",
                    "text-size": 8,
                    "text-font": ["Open Sans Semibold", "Arial Unicode MS Bold"],
                    "text-offset": [0, 0.6],
                    "text-anchor": "top"
                }
            });

        }
    };
    xhr.send();
});

We don’t want to stop here and make the visualization gods furious. Visualizations without click interactions is blasphemy. Since it’s customary to throw in interactions, how about displaying prison information in a popover.

// When a click event occurs on a feature in the places layer, open a popup at the
// location of the feature, with description HTML from its properties.
map.on('click', 'prisons', function (e) {
    let prison = e.features[0].properties,
        popOverContet;

    for (prop in prison) {
        popOverContet += prison[prop] + '</br>';
    }

    new mapboxgl.Popup()
        .setLngLat(e.features[0].geometry.coordinates)
        .setHTML(popOverContet)
        .addTo(map);
});

Putting all of it together, we get a map with prisons.

Although the end result looks impressive, its still difficult to answer a basic question — Where does the majority of prisons reside? West coast or the East coast? Heatmap is the answer.

To overlay a heatmap, create a “heatmap” layer and define paint properties.

//Add a geojson point source.
//Heatmap layers also work with a vector tile source.
map.addSource('prisons', {
    "type": "geojson",
    "data": dataSource
});

map.addLayer({
    "id": "prisons-heat",
    "type": "heatmap",
    "source": "prisons",
    "maxzoom": 9,
    "paint": {
        //Increase the heatmap weight based on frequency and property magnitude
        "heatmap-weight": {
            "property": "mag",
            "type": "exponential",
            "stops": [
                [0, 0],
                [6, 1]
            ]
        },
        //Increase the heatmap color weight weight by zoom level
        //heatmap-ntensity is a multiplier on top of heatmap-weight
        "heatmap-intensity": {
            "stops": [
                [0, 1],
                [9, 3]
            ]
        },
        //Color ramp for heatmap.  Domain is 0 (low) to 1 (high).
        //Begin color ramp at 0-stop with a 0-transparancy color
        //to create a blur-like effect.
        "heatmap-color": [
            "interpolate",
            ["linear"],
            ["heatmap-density"],
            0, "#f7f7f7",
            0.2, "#d9d9d9",
            0.4, "#bdbdbd",
            0.6, "#969696",
            0.8, "#636363",
            1, "#252525"
        ],
        //Adjust the heatmap radius by zoom level
        "heatmap-radius": {
            "stops": [
                [0, 2],
                [9, 20]
            ]
        },
        //Transition from heatmap to circle layer by zoom level
        "heatmap-opacity": {
            "default": 1,
            "stops": [
                [7, 1],
                [9, 0]
            ]
        }
    }
});

That brings us to the end. But wait, how about crime rates and its correlation with prison locations? Aha! That’s your take home. Hint — Download open data from catalog.data.gov for crime and assess relation.

References:

https://www.kickstarter.com/projects/237731198/the-book-of-openstreetmap
http://wiki.openstreetmap.org/wiki/Category:Software
https://github.com/tyrasd/overpass-turbo
https://www.mapbox.com/mapping/osm-data-model/
http://wiki.openstreetmap.org/wiki/Map_Features
https://overpass-turbo.eu/
http://wiki.openstreetmap.org/wiki/Map_Features#Vegetation_or_surface_related
http://wiki.openstreetmap.org/wiki/Overpass_turbo
http://osmcode.org/docs.html
https://github.com/osmcode/node-osmium/blob/master/doc/tutorial.md

Originally published at javascriptstore.com on December 15, 2017.