An illustrated introduction to APIs

Foreword

Have you ever heard things like:

- “Let’s use an API to show maps!”
- “We cannot accept payments because the API is down!”
- “If only we had an API to get this data automatically, instead of using a manual export!”
- APIs, APIs, APIs…

This post is an attempt to explain what APIs are — from scratch. In part 1, we look at APIs from a bird’s-eye-view. Part 2 will look at URLs in detail (i.e. since that's what we use to talk to APIs). Part 3 will show a practical example of using APIs.

This is currently a two-parts article:

• Part 1 // An illustrated introduction to APIs (this article)
• Part 2 // API Whispering 101

Introduction

How can we share data efficiently and securely? Not just with a few people that hold the keys to the database, but with a much wider and more diverse range of users.

This question has been at the core of most organizations’ strategic projects for years and “APIs” are part of the answer. If you find the term very confusing, this post is hopefully a chance to understand it.

API stands for Application Programming Interface. Note that in this post, we will focus on “Data APIs”, or APIs used to share and manage data. Towards the end, I’ll make sure to mention examples of other kinds of APIs. For now, you can assume that, at their core, APIs are software–sometimes big, sometimes very small–that act as middlemen between your database and your data users.

Let’s start with a story.

Once upon a time, in a little village, there was a single well…

With APIs, users don’t connect to the database directly. They use the access points provided by the API to interact with the database.

Imagine a village getting its water from a single well. Only one source for people with different needs:

• Citizens need to get drinking water for themselves
• The blacksmith needs water to cool his metal
• Farmers need water for their plants…

In the morning, the queue at the well gets crowded with people. Since the well was never meant to be accessed by everybody, panic can quickly ensue.

Now let’s imagine that somebody develops a network of water fountains connected to the well. Some fountains are optimized for individuals, some for professionals, some for animals. The situation is now much more manageable.

If your data is water and the database is the water well, then the API is the fountain network.

If your API is good enough (and that is why companies can have endless discussions about API design), you might even place fountains outside of the village so that inhabitants from other villages can access them. Selling water (data) could become a new source of income.

What comes out of APIs

What do we get when we work with an API?

We use the analogy of water above. Note that we are still providing plain water streams to users. We are not converting the water stream into bottled water before making it accessible, even though it would be convenient for thirsty citizens. If somebody wants to repackage what comes out of the fountain, they are welcome to do so, but APIs usually provide data in a form close to the raw data.

Remember that the road to hell is paved with good intentions. Every time you try to make your data easier to consume for one part of your clients, you make it more difficult for the other: bottled water would not work for farmers. And looking ahead to the future, nobody in the village would be likely to invent the washing machine if we only provided bottled water.

If your API is providing beautifully formatted PDFs, this might work well for a human wanting to read the content but the API becomes quasi-useless when you try to develop programs that use it as their data source.

So APIs provide data usually in a raw format. Don’t imagine an endless string of 1s and 0s though. What you will see as a user is mostly text files, usually in the .JSON format. That format is very easy to use in code, and still somewhat readable by humans, so clients are free to develop anything they want with it, including PDFs or water bottles.

This is what a .JSON file containing your client data might look like.

If you want some examples of API outputs, you can try to open these links in your internet browser:

• https://www.metaweather.com/api/location/44418/ will give you London’s weather in a .JSON format.
• https://freegeoip.app/json/ will give you information it can find about your ip address (i.e. the address of your computer on the Internet)

So, from the end-user perspective, APIs are special URLs that you can query to get raw data output.

Two types of UIs

To understand why APIs are called “APIs”, it is useful to know another acronym: GUI. GUI is API’s first cousin and stands for “Graphical User Interface”. You might not know it but you have been using GUIs for years. Both are API and GUI are interfaces, but designed to serve different customers.

APIs and UIs are not mutually exclusive, see them as two sides of the same coin. They are two different ways to display and share data.

On the one hand, GUIs are used directly by humans. They are called “Graphical” because they use visual elements to guide us: buttons, images, forms… Your spreadsheet software has a GUI and your favorite website too. Games are GUIs too. If you are not a developer, most programs you interact with on a daily basis have a GUI.

On the other hand, APIs are used in the code written by humans. Being interfaces for code, there is nothing to see, except lines of code. This doesn’t sound very visual, making it hard for newcomers to picture. But this minimalism is actually one of their main strengths.

Imagine that you are writing a program that needs to know the weather automatically, would you rather:

Option A

Use the Weather Company API that gives you a simple URL https://api.weather.net/geneva to call and sends you a text answer back like "sunny". Note that APIs are called via URLs, like websites.

You can actually try this for yourself with the Metaweather API which doesn’t require an account. As said above, open the url https://www.metaweather.com/api/location/44418/ in your internet browser to see London’s weather in a .JSON format.

Option B

Or write a program that would try to use the Weather Company website on your behalf by automatically repeating a series of interactions that were originally designed for humans:

• open an internet browser like Firefox,
• navigate to the Weather Company website,
• find the page for today’s forecast,
• find the little picture that describes current weather,
• analyze with some machine learning algorithm if it’s a sunny or a cloudy sky

Same data, different ways to access it. Personally, I would choose option A if I needed to write the code.

Before going any further let’s review these ideas with another story.

The supermarket analogy

The concept I really want to make clear here is that connecting to APIs with your program just another way to access services that you often already know via their websites.

Let’s see one more analogy. When you go to the supermarket, there are two entrances: the customer entrance and the service entrance.

The customer entrance is where you see products beautifully aligned in shelves. It is shiny, colorful and optimized for individual needs. This could be seen as the website of a modern company, using the storage warehouse as its database.

However, the customer part of the shop quickly becomes a nightmare if you want to do more than what it is meant for (e.g. shop for 100 guests at your dinner party). This is why there is another entrance: the service entrance.

The service entrance is often hidden from sight and definitely less pretty. It is optimized for delivery trucks. Products, which also come from our warehouse (i.e. database), are sold to you in wrapped palettes. If you want to buy one single bag of crisps it is not for you, but if you are trying to develop a restaurant (i.e. an application) this is definitely the entrance that you will favour.

We now understand why you need some kind of optimized-for-scale access to data…

But isn’t that what databases are for?! If someone needs that kind of access, you just give them access to the database!

Without APIs: data access issues

Letting people access your data directly can raise major issues.

1. If the data is sensitive, who should have access to the database?
2. How can we give different permissions to different people? Maybe most people should not get the permission to edit the database (read-only)?
3. If the data is heavy, how can we allow for different amounts of bandwidth to different people?
4. How can we precisely format and filter the data coming out of the database for different users?
5. When we migrate our database setup to a new one, how do we retire the old setup without crashing all the applications developed on top of it?
6. …

Some of these issues are just out of scope for traditional database software (e.g. issue 5.).

Others could already be tackled directly in database software (e.g. issues 1., 2.), with the caveat that the built-in solutions scale poorly and were never designed for a world where everybody uses the data. They were designed for a time when only a small number of database administrators (DBA) had access to the data.

Database software packages (e.g. PostgreSQL, MongoDB) are still very performant at storing/querying data and we want to keep using them for that. However, they don’t provide the level of tweaking required for modern sharing needs.

So the API is a layer of code that sits between the database and most database users. Another advantage is that since they are not part of the database software, APIs can be developed in any programming language. For the cost of managing this additional layer, you can use the full expressiveness of a programming language to set your data access rules.

How APIs help us internally

So how will this API layer help us? Before APIs, each team needed to develop their entire data management stack, even if they were extremely similar.

An classic example is web/mobile development. Not having an API might have been ok when you had just one website. But companies now want several websites (e.g. desktop and mobile), as well as an Android mobile app, as well as an iOS mobile app… With APIs, the data layer can be developed and managed only once by a small team. Then each team only focuses on the presentation of the data in its platform.

Web and Mobile development are fields where APIs have proven their worth, as is Data Analysis. In companies, more and more people need to interact with data on a daily basis. By providing an easy to use API, IT teams can service an ever-growing number of users, without letting too many people inside the database.

How APIs help us externally

Let’s first see where the old approach fell short.

In a world without APIs, each company mostly works with its own stack. You can see them as independent islands trying to do everything by themselves.

In an API world, the API layer can be set up so that it also provides access to external clients. Thanks to APIs, companies can make part of their data accessible to a much larger ecosystem.

Now, even if the idea of APIs starts to make sense, you might still be wondering why sharing your own data with the world can ever valuable?

Direct financial returns of sharing data

If you work for a for-profit company, the simplest way to benefit from sharing your data externally is to make people pay for it (with money or with their own data). This has been a business model for many years for Weather Forecast or Stock Market Data companies. Providing easy, on-demand access to their data is a new way to monetize their research. This lets them focus on their core business (creating the data) while others can concentrate on the display of the data.

Another example is maps. If you want to display maps on your website, you don’t need to buy a satellite and start taking pictures of the globe. Instead, you buy access to map data via an API from one of the many providers in the market. Thanks to APIs, you can use the data from professionals in many fields.

Indirect returns of sharing data

By improving their product with your data, other companies can also generate more customers for you. UBER, the transportation company, is a good example. Its API provides easy access to the UBER data (prices, drivers available, …). Lots of companies (Hyatt hotels, Opentable, United Airlines…) decided that it would be great for their customers to get easy access to UBER data, so they connect to the UBER API and offer UBER data on their app or website. You can find other examples here.

Does this mean that lots of people may not go directly to the UBER website to book a driver? Yes. Is it ok to sacrifice a bit of website traffic in order to get a lot more customers? Often, yes.

Sharing services

APIs are a useful way to share data. At the beginning of this article, we said that we would focus on “Data APIs” (i.e. APIs to share data). We also mentioned that they can be used for other things.

For example, APIs are also excellent to share services. Just like websites are used to share information and services with internet users, APIs can be used to share data and services with developers.

Modern applications have a wide collection of features that customers have learned to expect: payments, communications, navigation… These features are almost never developed in-house. Reinventing the wheel in these cases would take too long and cost too much. Instead, companies focus only on what makes their application truly unique and use APIs for all the rest. These API services don’t seem obvious when you don’t come from a programming background, but they aren’t abstract at all. Here are a few of them:

Payments

Let’s say that you want to develop a game in which users will be able to pay you. Would you rather focus on the game or start negotiating with banks to get a payment system in place?

It turns out that services like Paypal and Stripe provide APIs that let you process payments in your app with very little code. They have made their services available through APIs so that other developers use them in their applications.

Communications

Let’s say that you want to build a health assistant application. One of the features of your application could be to send a text message to a person of your choice if you haven’t taken your medication in a long time.

If you tried to do it from scratch, programming just this text messaging feature could take you years of development and negotiations with network operators. However, you can also simply use Twilio, an API that does text messaging for you as long as you send it a well-formatted .JSON object.

If your Twilio account ID is 1243 and you want to send a text to number 077 777 77 77 that says Hello, it would be as simple as sending an API request like this:

POST https://api.twilio.com/Accounts/1234/Messages.json \
--data-urlencode "To=+0777777777" \
--data-urlencode "Body=Hello"

This syntax might still look unfamiliar but note how little code you need to write. As long as you have the right URL, the rest is only data, data specific to your text message (a phone number and a body content). In return, Twilio would just charge your account for the price of a text message. Since they can provide numbers from all over the world, you can manage communication with your clients as if you had a call center in every country.

Navigation

We have mentioned maps before. However, the story doesn’t stop with sharing maps (data). Map providers (e.g. Mapbox or Google Maps) also offer on demand turn-by-turn navigation from their APIs, letting you add navigation features (e.g. “Be guided to our nearest shop”) to your application with very little code.

Conclusion

We have seen where APIs sit in the technological landscape and why they are considered so useful. They allow us to share data and services, both internally and externally. Since they use basic URL schemes as their interface, you don’t have to understand what is happening inside the API to benefit from their services. In my next post, we will look at URLs in detail since they are the most common way to interact with APIs.

About the EPFL Extension School

The EPFL Extension School teaches digital skills in data science, machine learning, web application development, UI development with React.js, and more. Please visit our website, reach out to us by e-mail or Twitter, and most importantly: go get those digital skills!