API Whispering 101

Foreword

If you haven’t read our first article An illustrated introduction to APIs, and don’t know what an API is, it might be easier to start here. In this article, we take a closer look at what an API request is.

• Part 1 // An illustrated introduction to APIs
• Part 2 // API Whispering 101 (this article)

Warning There are several possible architectures for APIs, which in turn require different types of requests. Presenting them all would turn this article into a heavy book. To keep things approachable, this article will focus on API designed according to the REST architecture. Just be aware that there are other API architectures out there, some older (e.g. SOAP), some newer (e.g. GraphQL), that we do not cover here.

Also, this is not really a tutorial teaching how to program with APIs in depth. Here the goal is to teach some of the underlying logic that is so often skipped in tutorials.

Introduction

Using an API and creating an API are two very different things. You could use APIs for years without understanding how they work under the hood, and there would be nothing wrong with this. In this article, we aim to cover the minimum knowledge one needs to use APIs.

APIs are usually hosted on web servers. You can think of web servers as computers without screens (i.e. headless computers) that can only be reached via the internet. You interact with an API by sending a request to the web server and the server sends a response.

Requests and Responses are two really important words in the world of APIs:

• I send a request to the server
• The server sends me back a response
• I send another request to the server
• The server sends me back another response
• …

APIs give different kinds of responses (i.e. “data”) when they receive different kinds of requests.

Both requests and responses are text files and are written according to a well defined convention: the HTTP protocol.

If the HTTP acronym sounds familiar, it’s because you use this protocol without knowing it every time you write website addresses (called a URL) in your internet browser (i.e. the program you use to navigate the internet). Remember how website addresses start with http:// or https://?

Yes, you actually use web servers and APIs every day, every time you visit websites.

This is something that trips up many beginners, as everybody assumes that sending a request is way more complicated than writing a URL. However, it doesn’t always have to be:

• typing a URL in your internet browser is one type of request
• and getting back a HTML web-page is a type of response

When you think about it, the most common job of an API is to serve data… and web-pages are after all one kind of data (i.e. data that your internet browser displays in a pretty way).

So APIs and websites should not be seen as two opposing things. Most websites are just one output (i.e. the HTML output) of an API: an output designed to be seen by humans in an internet browser.

If you read our previous article on APIs, you will remember that we used the analogy of a supermarket with its customer facing area (i.e. the website) and its warehouse (i.e. the API). There as well, the warehouse is the back-end that is used both to fill the shelves area and welcome professional customers in need of bulk orders.

So, where do we go from here? If we understand what a request is–this mysterious HTTP compliant text file–we will be able to talk to APIs.

A first look at raw request files

As said, both the requests I send and the responses I receive are just text files. Although you can write pretty much anything you want in a text file, you will have to observe lots of rules if you want to write valid API requests.

Respecting these rules might sound like a lot of work but the process is mostly abstracted away from us. When we use an internet browser, we can simply provide the URL and the browser will write the entire request’s text files for us in the background.

Same thing when programming, we usually don’t need to write the entire request text files or to parse response text files all by ourselves: a library will do it for us.

Still, it is useful to know what requests and responses look like. Below is a very minimal request’s text file:

GET /index.html ➊ HTTP/1.1 ➋ 
Host: www.example.com ➌ 
Accept: text/html ➍

The first line contains the URL's path (/index.html ➊) and the HTTP protocol version (/1.1 ➋). The following lines (➌, ➍) are called HEADERS and are written like this:

Header-name: Header value

You can think of headers as metadata about the request. Here for example, our two headers mean that:

1. The Host: header means that the path we are looking for (described in ➊) is on www.example.com
2. The Accept: header means that we want the server to send back a response in the html format

One thing bothered me when I saw my first request: what I considered a full URL (i.e. http://www.example.com/index.html) looked split between three different parts (➊, ➋, ➌). I think it would be more visual if the full URL was in one place but that’s not the way requests are designed… Luckily in code, just like in an internet browser, we have tools to do the job for us.

Here is another request, slightly more complex. This one is a POST request (i.e. starting with the word POST instead of GET).

POST /Geneva HTTP/1.1
Host: api.twilio.org
Content-Type: application/json
Content-Length: 278
{"message": "hello"}

Note that POST requests, unlike GET requests, can have additional data after the HEADERS. Here the additional data is {"message": "hello"}.

What about response? Again, the HTTP protocol is about simple text messages, so responses, just like requests, are basic text snippets. The convention is just a bit different. Here is a simple response that you could receive from a web server:

HTTP/1.1 200 OK
Content-Length: 88
Content-Type: text/html
(more HEADERS...)
<html>
<body>
<h1>Hello, World!</h1>
</body>
</html>

This response also comes with some headers, followed by some content. Here the content, starting after the blank line, is a very basic web-page written in HTML. But it could have been any kind of data (e.g. financial records, basketball photograph…). If you don’t know what HTML is, think of it as the language in which web-pages are written.

When you type a URL in your internet browser, a request is generated and this kind of response is received. As a user you only see the HTML content after it has been “rendered” (i.e. transformed into a pretty web-page). Nearly all web-pages that you see daily are made of HTML, including this article.

Not convinced this is happening when you browse the web? Let’s see…

Modern internet browsers (e.g. Mozilla Firefox, Google Chrome…) actually let you see what is happening under the hood. In Firefox, I can press the keyboard shortcut Command-Option-E (on Mac) / Ctrl-Shift-E (on Windows) and see the Network Monitor window, which records all the requests being sent and all the responses being received.

If I go on a website (e.g. www.wikipedia.org), open the Network Monitor, and start clicking on links, I will see a long list of HTTP events happening. Clicking on one of these events will show me all the details, including headers.

Now, personally, rather than thinking about requests/responses as raw text files, I always picture the entire business of sending requests/receiving responses as managing a Post Office on my laptop.

Congrats! You are now managing a Post Office!

Since I find the Post Office analogy more visual, here is how I like to imagine these requests.

On the front of the envelope/postcard/package, there is:

• an address (e.g. URL like www.extensionschool.com)
• a stamp (i.e. the HTTP verb that you want to use — we will explain this soon)

On the back of the envelope, there is some metadata (i.e. some of the header fields).

Inside the envelope, there might be some mail content. The envelope can contain data or it can be empty, depending on the stamp you are using and what you want to send to the web server. As said before, GET requests cannot carry data outside of their headers, but POST requests (and a few others) can.

Raw request text files can really look daunting, but they don’t have to be if you think in the context of an envelope. Let’s look at each element. There are 4 things you need to understand about API requests:

• TheURL (i.e. the address)
• TheHTTP verb (i.e. the stamp)
• The optional content of the request
• The headers

Address / URL

The address is the most straightforward element because you are already using them every day in your internet browser. However, we saw that the address is cut into smaller parts in the request file, so it is important to understand the name of each chunk.

We often try to describe the Internet as if it was a street where all resources (websites, data APIs…) have an address. From now on, we will call these addresses URLs.

There are five parts that make modern URLs:

1. Top-level domain
2. Sub-level domain
3. Path
4. Hostname
5. Parameters

Top and Sub-level domain

Some URLs are fairly simple. They just contain a Top-Level domain and a Sub-Level domain. Top-Level domains are what you know as the big families of websites (e.g .org, .com, .ch). Sub-Level domains are what most people call “website names” (google, apple, epfl). URLs that only contain a top and a sub domain point to the main door of the relevant buildings:

• wikipedia.org : Wikipedia homepage
• google.com : Google homepage
• letemps.ch : The Swiss newspaper Le Temps homepage

Path

Other URLs are more precise. They take you to a specific part of the website or you could say that they point you to a specific floor and room in the building. To do that, some website just add a Path (i.e. words separated by /) after the Top-Level domain. This is a bit similar to the path for your files in your computer.

• https://www.sbb.ch/fr/abonnements-et-billets.html : the ticket shopping page of the Swiss National Train company website
• https://www.lego.com/en-us/products : the product page of the LEGO website

Note that allowed paths are often called endpoints in API documentation.

Hostnames

Sometimes, especially if they want to divide their website into big chunks, webmasters will divide sites under several Hostnames, which is the part that comes before the Sub-Level domain:

• https://en.wikipedia.org/ : The English version of wikipedia is under domain en.
• https://fr.wikipedia.org/ : The french version of wikipedia is under domain fr.

Websites that don’t set up multiple hostnames usually only use the most famous hostname www. To reach the level of granularity you need, you can even set up your site to use both hostnames and path:

• https://en.wikipedia.org/wiki/Massive_open_online_course : The MOOC page on the English version of Wikipedia
• https://exts.epfl.ch/courses-programs/50-things-you-need-to-know-about-data : The page describing a specific course (50-things-you...) on the EPFL Extension School hostname (i.e.exts.) of the EPFL sub-level domain (i.e.epfl.ch).

Parameters

Some URLs store even more information after the path. Can you spot the difference in structure between the two URLs below?

1. https://en.wikipedia.org/wiki/Massive_open_online_course
2. https://www.google.com/search?q=pandas+sneezing&tbs=vid

URL 1 is a plain URL, containing a hostname, top and sub-level domain and a path. URL 2 also contains a typical structure, from the https: to /search?, but the path is followed by two parameters (i.e. what comes after ?). In this case the parameters are:

• q (i.e. query) is set to pandas and (+) sneezing
• tbs (i.e. to be searched) is set to vid (i.e video)

This is a Google search for sneezing pandas videos. On Google, users can search for anything. As granular as hostnames and path can be, Google doesn’t want to store a page for all possible searches (that would be trillions of pages).

So they use a different strategy:

1. The user writes keywords in the search box of the Google homepage
2. The page grabs the written keywords and uses them to create a custom URL
3. A request is sent to the custom URL
4. On the server, all incoming requests are analysed and the ones starting with https://www.google.com/search? get special treatment: instead of returning a page that is already saved in memory, the parameters are extracted and used to create a custom result page, just for you, at the time of the request.

So first they create these dynamic URLs from what you enter in the Search box, then the web server analyses the different query parameters in the URL and creates a custom result page for the user on demand.

It’s one of the great features of APIs: you can configure them so that they listen not only for precise URL calls, but also for fuzzier URL calls (i.e. "https://www.google.com/search? and anything coming after").

The choice of q for query and tbs for to be search is entirely up to the website owner. When creating your API, you could define your very own set of accepted query parameters. If your application contains cooking
recipes, you could have a parameter called ing= for ingredient and another d= for diet.

Because nothing dictates how parameters should be named, one of the first tasks when learning a new API is to discover what parameters it accepts. This is typically the type of information you will look for in an API documentation.

Another advantage of parameters over paths is that they are not order sensitive.

URLs in practice…

In practice, developers don’t split URLs in lots of different parts. In your internet browser, you just type a link in the URL box and you are taken to the website. Same thing when coding, let’s say that I want to:

• Find all the train connections between Geneva and Zurich with the free API of the public train service in Switzerland
• The API URL is:

https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich

• Programmers don’t write a text file in which they split opendata.ch from /v1… Instead they just give the full URL to a function (i.e. here GET()) that does it for them. In the programming language R, it would look like this:

library(httr)
GET("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich")

• GET() will write a valid request file, send it for us and give us back the received response
• And we would get a fully detailed data object that we can work with:

{
  "connections": [
    {
      "from": {
        "station": {
          "id": "8501008",
          "name": "Genève",
          "score": null,
          "coordinate": {
            "type": "WGS84",
            "x": 46.210237,
            "y": 6.142422
            ...

Stamp / HTTP verbs

When you call an API's URL, you can do it with different types of calls. These types are called HTTP verbs. We saw GET and POST, the most common, but there are others: PUT, DELETE… When the API receives your request, it sees which type of call you used and can answer to you in different ways.

I like to think of the HTTP verb as the stamp: right there at the front of the envelope, next to the address. You do need to use different stamps if you are mailing a postcard or a heavy package. Similarly, different HTTP verbs allow for different request content.

• GET is mostly used for “postcards” that cannot encapsulate content, outside of the URL and the headers
• POST and PUT are a bit more fit for “packages”, or “letters in big envelopes”

There are best practices about which verb should be used for what, but as an API user, you will just trust the API documentation to tell you what you need for what action.

HTTP verbs in practice…

In practice, changing the HTTP verb is usually as quick as calling another function for programmers. We saw before that we could grab the train connections between Geneva and Zurich by using:

library(httr)
GET("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich")

If the API requested POST instead of GET, we would have done:

library(httr)
POST("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich")

POST(), just like GET(), are functions that write a valid request file, send it for us to the API and give us back the received response.

The main difference between the two is that GET() helps us manage GET requests, and POST() helps us manage POST requests. These functions are coming from the library httr and have been named by its authors after the HTTP verbs to make them easier to remember, but you could find other libraries that do the same thing where the function for GET requests is named get_request(<enter-your-url-here>) or where the function POST for requests is named httpRequestPost(<enter-your-url-here>) : these are just the function names.

Mailed content

As said above, different verbs have different properties, which we won’t cover in detail. If you have been able to navigate websites without ever thinking about verbs, it is because your internet browser uses the needed HTTP verbs in the background.

On the one hand, some verbs, like GET requests, are like postcards and cannot enclose additional content. The URL (including the path and the query parameters) and the headers are the only place to store “data”. Using parameters allows us to create an endless number of unique URLs, so the absence of enclosed content is not necessarily limiting in terms of quantity. But it can have shortcomings…

Reasons why you might want to store data outside URLs

• it can lead to very long URLs. There are times when you want to send entire sentences to the server…
• all the data is visible in the URL. There are times when you want to send sensitive data to the server…
• you are limited to parameter=value patterns and cannot use more complicated data organization, like nested parameters…

On the other hand, if you use POST requests, it’s a bit like mailing a package, since POST can contain data outside the URL and headers.

Don’t get me wrong: this doesn’t make GET requests useless. When you just want to visit a public web-page, GET calls are just fine. If I send a GET request to www.wikipedia.org, I will get back Wikipedia’s homepage as a response. Just sending a GET request to this URL is enough for www.wikipedia.org API: no more data that is needed.

The data stored under the headers (i.e. inside the envelope) is called the request’s body, or sometimes payload.

Request content in practice…

As with verbs, we could send a POST request with:

library(httr)
POST("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich")

When APIs ask you to use POST requests instead of GET requests, it is often because they need you to send additional data that you cannot put in the URL. In code, the POST() function can accept more data if you give it after the body= argument.

library(httr)
POST("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich",
     body = list(name="Caroline"))

Again, a valid POST request text file is written for you, including whatever data you gave to body=.

Metadata

Some metadata is sent with every request and every response. The metadata is sent in the form of a list of colon-separated name-value pairs: they are called Headers.

Headers can be used to pass a lot of information:

• Information about the content we want to get back (format, language, platform)
• Information about who is making the requests and from which site the person is coming (if you clicked on a link)
• Information about the session (e.g cookies)

Most of these are set automatically for you though and lots of API users never mess with headers. If you are curious about what kind of information can be shared via headers, you should have a look at this great illustration by Julia Evans.

On our little envelope illustrations, we showed the Accept header, which lets you say what kind of data you want to get back. Most API’s endpoints can only serve one type of data, so setting the Accept is not always required. But some well designed APIs let you choose among many.

Request headers in practice…

Again, most of the work is done by the libraries used by programmers. All headers already have sensible default values so you only change the ones that need special value.

library(httr)
# Using all default values
POST("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich")
# Requesting for .CSV content (if supported by the API)
POST("https://transport.opendata.ch/v1/connections?from=Geneva&to=Zurich",
     add_headers = list(Accept="text/csv"))

The data given to add_headers= will be properly converted to well formatted headers in the sent request file.

In practice — Recap

So how does it look like in practice? Do programmers write things like this:

GET /v1/connections?from=Geneva&to=Zurich HTTP/1.1 
Host: transport.opendata.ch 
Accept: application/json 
...Long list of additional headers...

Most don’t. In reality, programmers use libraries (i.e. bundles of code written by other programmers) that provide functions. These functions do most of the work for them. This article is not an introduction to programming with APIs so we have only shown brief examples that give an idea of the code.

For example, if one wants to:

1. send a GET request to the URL: http://www.example.com/index.html
2. save the response into a variable called my_response
3. extract the HTML content stored inside my_response and save it into another variable called my_html

In the programming language R, it would look like this:

library(httr)
my_response <- GET("http://www.example.com/index.html")
my_html <- content(my_response)

• First we activate the httr library which will let us use the GET() function.
• Second we use the GET() function which only asks for a full URL and handle the writing of a valid request
• Third we extract the content of my_response with another function from httr named content()

Similarly in the programming language python, in which the code would look like this:

import requests
 
my_response = requests.get("http://www.example.com/index.html")
my_html = my_response.content

• First we activate the requests library, an equivalent of R’s httr but for python users, which will let us use the get() function.
• Second we use the get() function which only asks for a full URL and handles the writing of a valid request
• Third we extract the content of my_response, by adding .content after

As you can see, even if you don’t understand the code, the patterns are very similar and the programmer doesn’t need to know the hundreds of possible HTTP headers, nor the strict grammar of request files.

Conclusion

To read the documentation of an API and start using it, there are four things you need to understand:

• The URL (i.e. the address)
• The HTTP verb (i.e. the stamp)
• The optional content of the request
• The headers

Then you use libraries to write the correct requests for you and parse the results into something useful.