Cities and APIs, Part 2

This is Part 2 of an ongoing series, “Cities and APIs”. Read Part 1 here.

Last time I wrote about Cities and APIs, I argued that modern day urban problems, especially urban inequity, cannot be solved without accessible urban data. But access isn’t enough — it’s what we do with it that matters.

Christoph Raetzsch (The Free University of Berlin) echoes this sentiment in a 2019 paper where he explores the role of APIs when making urban data available. City APIs often fulfill conflicting commercial and civic demands and creates a “zone of socio-political contestation”. For example, when a tech giant like Google works on a large-scale redevelopment of Toronto’s waterfront, we start to wonder whether these “smart city” aspirations preclude civic engagement through its very instrumentation, disregarding the political axes of history, justice, and culture in the process.

We need to scrutinize, or at least think deliberately about how we can get all stakeholders in a city involved and determine what purposes such an interface to data shall serve and for whom. Raetzsch proposes a vision in which APIs “allow citizens to tap into various seams of the social fabric in a city, to reveal submerged layers of histories, experiences, and identities in urban spaces.”

I believe that these understandings and the people who build on top of such insights can change our cities for the better.

In Part 2, I dive deeper into the weeds of using city API’s and answer questions that readers of Part 1 have asked.

Q: What kinds of APIs are there and which are relevant to cities?

API stands for Application Programming Interface. All APIs do one same thing: make certain data or functions in an accessible, standardized way so it can be used by other applications. Simply put, it allows two apps to talk to each other. By doing so, this makes APIs integral to any kind of web-based communication!

Literature suggests (which I mean by a quick Google search) that there are four main types of APIs:

1. Open APIs: Publicly available, minimal, or no restrictions to access.
2. Partner APIs: Often proprietary, accessible between business partners. Developers need rights or licenses.
3. Internal APIs: These private APIs that are for internal use, like a company internal system.
4. Composite APIs: Combines different data and APIs to get information in a sequence of tasks.

But there are other ways you can categorize APIs other than the level of access.

For example, Web Service APIs are a subset of APIs. In general, when we speak about APIs, we are likely speaking about web APIs, which is are APIs accessible over the internet. To further break it down, a popular type of web APIs is a REST API, which uses a standardized style to utilize HTTP methods to create requests over a network.

In addition, Parth Detroja, in his book Swipe to Unlock talks about 3 kinds of APIs:

1. Feature APIs: Lets an app ask another app to do something it can’t do. For example, the Uber app borrows the help of Google Maps API to show destinations and the help of Paypal’s Braintree API to process payments.
2. Data APIs: Lets an app ask for information; for example when a certain train in New York’s subway system will be arriving at a station.
3. Hardware APIs: Lets an app access features of certain hardware; Instagram uses the iPhone’s Camera API and Geolocation API to physically take photos and know where it was taken through the sensors.

Given these buckets, when we talk about increasing access to city APIs we are probably talking about Open APIs, Web APIs, and Data APIs.

There are other relevant kinds of APIs in a city. Look at Coord, a Google-backed platform that sells a service to cities and companies that offers a searchable inventory of every curb in a city and a way to analyze them in real-time. Using physical sensors and datasets to know when certain space is available in a curb (bus stops, loading zones), it can help people route a seamless trip experience from A to B. Here, you can start to see a network of data APIs, hardware APIs, partner APIs, web APIs, etc. working together to make this platform happen.

This means that APIs increasingly function as a gatekeeper of data and a new kind of infrastructure in the urban realm. For example, say I created an app with a proprietary API that helps garbage trucks pick up curb trash at the right times to avoid congestion. Here, the API served as 1) an infrastructure that changed how the truck navigated an urban space AND 2) a gatekeeper of data that is tightly controlled by whoever develops and owns the API.

Q: What are the hurdles facing developers and others who build apps for cities?

In Part 1, I explained how in cities, APIs are housed in silos across city governments, making it difficult for developers to make sense when data is needed from multiple urban functions. These issues in structural access pose friction points when building apps.

There are a couple more unique challenges facing developers, namely regulatory hurdles and lack of incentives.

Let’s imagine you are a developer that wants to build an app for cities. You don’t work for the government, nor do you work for a large institution like Google. In a year, you have a startup that’s the garbage picker-upper app mentioned above. But your app has been awaiting approval for a pilot in the city for months and is making no money. Would you reconsider?

Heavy regulation such as lengthy procurement and a lack of opportunity to test and iterate their product prevents developers hinders the whole development process. Even if you create an amazing app, it could take at least 2–3 years to have multiple pilots in a neighborhood to see if it works.

Scenarios like these turn off developers, entrepreneurs, and investors from urban tech. While there is huge potential for sustained public value and positive externalities, on the flip side it is difficult to monetarily capture some of the value you create. For small companies starting out, sometimes, the costs are not justified.

Q: What are some accessible city-related APIs that are out there and how do you start using them?

Last time I simply just gave a starter list of urban-related APIs, but here’s more context!

Turns out, the “siloed data” and the “lack of accessible API’s” problems are not some profound and recent secret in city governance. Even better, since around 2010, state and local governments have poured millions of dollars in publishing datasets, building API portals, and investing in IoT (“Internet of Things”) infrastructure. The Sunlight Foundation reports more than 100 cities with open data policies with data availability in key areas like crime, zoning, budget, etc. RList Insights compiled a list of 200 major cities around the world and their open data portals.

This big movement in open data shows cities making an effort to help government employees access cross-departmental data, help foster innovation, and bolster public trust through transparency. For example, Barcelona’s Apps4bcn project serves as a hub of apps and city services that connects developers and sector experts in tourism, culture, healthcare, education, etc.

With this movement, 3rd-party institutions like the Open Data Institute and GovEx have also emerged to help cities better manage these projects to maximize civic impact. Private companies have analyzed the economic impact as well, as the consulting firm McKinsey has estimated open data’s economic potential at more than $3 trillion globally.

This growth is enabled by companies like Socrata, which helps cities set up open data programs and websites. Socrata hosts over 100 open data portals and you can peruse them on their Open Data Network.

However…are these Open Data efforts working? Depends on who you ask, and what the goal is.

Some argue these efforts may not justify the hidden costs. A subscription to operate Socrata, Opengov, or Junar costs $10k-$50k annually for cities while LA spent $319k in startup costs alone, with $287k in annual expenses.

Some say while Open Data portals have succeeded in meeting the goal of availability by making a lot of information available. However, while a handful of “top-performer” cities have excellent portals, others have APIs that are incomplete or not actionable. Often these datasets are static, which means it could be outdated, without dynamic data like real-time occupancy of public spaces. These portals also vary greatly across departments and cities in terms of subject area and standardization, with often unstructured and incomparable data.

To have more clarity, at the end of the day, we have to go back to the purpose and function of a city. We have ask questions like, are residents healthier because of open data? Are streets safer for pedestrians? Are city governments more responsive to their citizens? The jury is still out.

Exploring City Data with Socrata with Ruby — Getting Started

Every Socrata dataset offers a corresponding built-in SODA (Socrata Open Data API). These standardized APIs make it easy to explore any city data on Socrata.

1)First, let’s create a new directory on your computer and add a new Ruby file. I will call mine “cities.rb” and open it up with VSCode.

2) We will use the HTTParty gem to help parse our JSON file into a hash datatype so Ruby can read it easily. SODA data is in JSON (JavaScript Object Notation) which is a format for structuring data in a collection of name/value pairs. HTTParty is one of many methods for doing this, but it gives us a simple process and convenient methods than many others. To install, run “gem install httparty” in terminal.

3) Then, type in “require ‘httparty’” into your empty document to be able to use the gem.

4) Now is the time to specify what data we’d like to explore. To do this, we find the endpoint of the dataset and define it within a method, like below. As an example, I am using an endpoint that gets us a list of alternative fuel stations in Chicago. This endpoint is simply a publicly available, unique URL that represents an object in a Socrata dataset. We can then send HTTP requests to this endpoint and receive a relevant JSON from the remote server.

5) Next, I write a custom ruby method that passes in our URL and utilizes httparty to parse what we receive from the server.

6) Now we are set to work with our data! Run IRB and load the file into it so we can work with the methods we just defined. Now, if you type in “my_url” the console will return the defined URL.

7) Similarly, we can pass in my_url into our response method, storing it in a variable “response”. This will return a huge collection of hashes and arrays. This means that everything works!

8) Now to just see the keys that are in our set of hashes, we can run “response[0].keys”. This will give us an idea of what categories of data describe each gas station.

9) From what we know about the keys, we can use a Ruby enumerable “.map” to get an array of all the names of gas stations.

10) Voila! Now you have an idea of how to get the data you want from a Socrata API! To explore more, check out the SODA documentation.