A new marketplace for civic data science
With scarce resources for existing programs, many new local government data science projects rely on low-bono graduate student interns by default. So we sought to explore how we could make those engagements as modular and streamlined as possible.
Over the coming weeks we’ll be bringing you firsthand accounts from graduate students at New York University’s Center for Urban Science and Progress describing the work they completed and the relevance for the future of local government operations.
Our first project builds on ARGO’s Street QUality IDentification (SQUID) work, adding a geoprocessing layer to pull data from Open Street Cam for any part of the world. NYU’s Charlie Moffett partnered with ARGO Data Scientist David Marulli to add a geoprocessing layer to pull data from Open Street Cam for any part of the world. Cheers, Patrick
Parsing open street cam imagery from any point around the world
By Charlie Moffett
Cities currently measure street quality by using low tech human surveys or high tech lasers attached to automated vans that plod every inch of the road. The former are subjective and the latter high tech solution is expensive. With open source imagery, however, we can now collect that same information using our mobile devices to create low-cost, “low-touch” ride quality maps at city-scale. Resources are still required to traverse the entire street grid, but that can be handled in part by leveraging the power of the crowd and utilizing any of the many city vehicles that already traverse the city streets.
An example from the City of Los Angeles demonstrates why this efficiency gain is meaningful. Condition assessments in LA were last conducted by the Bureau of Street Services in 2014, and were in fact quite precise at the time.
Here’s an image from their assessment for a section of road near downtown:
Surface conditions had deteriorated since then though. Below is a more recent ride quality survey of the same location, collected using ARGO’s SQUID platform:
When the Bureau caught wind of this new ground truth, they made short work of laying new pavement to rescue their scores there. Here’s that street again, surveyed just a few weeks later:
By capturing all of the surveys like these for a given city, we can begin to blanket entire street networks in ride quality data. The first step in bundling street quality data for a city is establishing its spatial geometry. OSMnx gives us the power to easily download and construct street networks from OpenStreetMap by simply providing a place name. Using the collection of nodes from a NetworkX graph defined by an administrative GIS boundary, we set up a geodataframe containing decimal degree coordinates for all of the intersections in the street network.
Because sensor data supplied by SQUID Bike and other efforts live on OpenStreetCam (OSC), we need a list of unique identifiers for OSC collection trips within our geometry. With the help of HTTP library requests, we send our coordinates to the OSC API and parse corresponding JSON responses for ‘sequence_id’s related to all trips that pass through or near the intersections. After some minor cleaning, we can now store the right records for our city of choice.
The last thing we did was to push the trip records to a spreadsheet through the Google Drive API. With gspread, it’s relatively straightforward now to read, write, and delete data from a Google Sheet using Python. By plugging our output into one of ARGO’s existing public data infrastructures — the Streets Data Collaborative (StDC) — we were able to visualize ground-truthed street quality data for the entire city network. In this case, Berkeley:
By utilizing open data and leaning on the great work done by others in this field, in just a few short weeks I was able to refine my wrangling and scraping skills and tackle this piece of the larger assessment and maintenance pipeline. I’ve also learned that the real value of developing public solutions for streets lies in the establishment of standards and shared practices, which enable ever-greater scales of knowledge exchange and torch passing.
In addition to ARGO’s street maintenance work, projects like SharedStreets are also building tools toward a public-private “digital commons” around transport data so that cities can get the data access they need to revitalize these invaluable assets. In the future, work from this winter break project could be bridged with existing SharedStreets ‘Reference ID’ standards to bring ground-truthed street quality measurements into the commons, inching ourselves that much closer towards the dream of improving how we measure the quality of and thus maintain our city streets.
Charlie Moffett’s code for this project can be found in the ARGO marketplace on GitHub.
