REST API Development tips and best practices — Part 3

Documentation tips and moving beyond the basics

This is the last in a series of three posts on REST APIs.

• Part 1: Introduction and planning
• Part 2: Schema suggestions, common mistakes and deprecation
• Part 3: Documentation tips and moving beyond the basics

DOCUMENTATION

As with any type of software project, good documentation goes a long way to increase adoption of your API more quickly.

The two most popular options in the Specification section in the first article of this series can come to your rescue as documentation options as well. If you have adopted the OAS, you already have a great solution for documentation and integrated manual testing of your API in the form of Swagger UI.

The Postman app provides similar features in this area, and as a last resort, you always have the option to generate your own documentation from scratch (although that will be more difficult to maintain and will lack an integrated testing environment).

1. Don’t forget the errors

While the resources and schema of your API can usually be easily determined by your consumers with scarce information and a few test requests, the errors that your API returns can be harder to discover. You need to document all of them in detail, in combination with the endpoints that can return them and/or the conditions that will cause them to happen.

Experience shows that errors are one of the first things in the documentation that teams fail to update to reflect changes in their code (when the workflow does not use a specification-first approach, such as OAS or similar).

2. Entity reference

If your application is big enough, chances are that your API returns multiple types of entities that can sometimes have complex relationships between them. Some of those entities will probably grow big enough to contain large numbers of data that you might not want to return in every call.

Field filtering might return a default filtered view of your objects that only lists a handful of their available attributes, thus not providing consumers with a full overview of each entity with a simple request.

In such cases, it’s preferable to have comprehensive separate index of available entities that developers can browse through to disambiguate the use of each field and relations between entities. Furthermore, if you don’t provide an SDK or a set of business objects, it‘s an excellent resource they can use to develop their own set of data models on the client.

3. Version history

This one is really important to make sure that your clients are always up to date. You may add non-breaking incremental updates to your API without having to deprecate your major version (making minor version increases), but you should communicate them clearly to your consumers.

You should maintain a separate documentation version for each new version of your API (even minor ones) still available in production. That way your consumers can keep track of your changes as quickly as possible and take advantage of them in the next version of their client apps, even if they skipped a few minor versions.

Readily available changelogs between versions are also great to provide a quick overview of all the updates.

MOVING BEYOND THE BASICS

As I mentioned in the introduction, this post does its best not to repeat common advice that you read in every other REST API development resource online. Based on that, I assume that you have read at least a couple of such resources before you got here and you have a clear understanding of the basic features your API needs to implement.

In this section, I will mention a few not-so-basic features that I’ve found very helpful over the years and don’t see mentioned all that often.

1. HATEOAS

Although Roy Fielding’s dissertation thesis states that there is no REST without HATEOAS, and people go so far as to call APIs that don’t support it RESTless, there is a growing number of people who find HATEOAS… problematic (mildly put).

Up until a few years ago, I took a strictly pragmatic approach to REST API design. Having to work on larger projects with long histories and terrible versioning problems made me reevaluate my approach. While I am still skeptical about whether the bloat of HAL or alternative Hypermedia formats make perfect sense, I do appreciate practical solutions such as GitHub’s use of hypermedia (although GitHub has moved to GraphQL for the latest version of its API).

I won’t claim I have a perfect understanding of HATEOAS and what all its touted advantages can do for an API in practice, but I have come to appreciate some of them. I haven’t seen the ideal client yet — one that works just by knowing the API’s base URL, but it may be because I haven’t searched hard enough.

2. Caching

The HTTP protocol has supported caching mechanisms from the early web years with varying degrees of success. Although the initial protocol standard could be improved, failures were often attributed to bad server configuration or inadequate implementations of servers or clients (including early web browsers).

We are nearing the end of 2018 and HTTP 1.1 is almost universally available, with HTTP 2.0’s adoption increasing rapidly, yet many Web APIs (not the ones from major companies) keep ignoring standard protocol caching mechanisms altogether. At best, clients will attempt to use their own caching approach (through implied parameters provided in the APIs responses) and at worst, there will be no caching at all.

The above situation sounds bad but is made even worse considering the whole protocol caching infrastructure works automatically for most clients (browsers, native networking libraries etc), as long as APIs return the proper caching headers! If that doesn’t sound like so much work, help improve the web by using caching in your API.

I won’t go into all the details regarding HTTP’s caching mechanisms since there are many resources you can read online, but if you know what caching means in general, implementing it in this context is much easier than you may think. All you have to do is let the client know whether they are allowed to cache a response and when you expect its contents to change (and therefore invalidate the cache).

If you want the client to cache a certain response, but you are not certain when it might change, don’t make them download the entire response body every time. Just use ETags and you’ll only have to provide a short reply that lets them know if the content they requested has changed or not, since the last time they downloaded it. Finally, unless you have to support really old clients, you can skip the HTTP 1.0 cache mechanisms altogether (although they should be trivial to implement if you got everything else right).

3. Rate limiting

Often, rate-limiting is one of the last things API developers implement, usually after there has been some sort of service abuse reported. I believe it should be an integral part of any API which serves more than a handful of users and should be implemented with care.

A common mistake developers make is to rely on naive implementations that have only been tested by a couple of people during development. Don’t be fooled! A simple algorithm that measures visits by an IP address in relation to time is likely to cause a massive denial of service for your new big B2B client, who use a limited IP space for their network clients! Always take into account authentication tokens and be very careful about how you implement rate limiting — or don’t do it at all before you have educated yourself and thought about all the possible consequences.

If your API does implement rate limiting, make sure to inform your consumers by providing their current status via response headers. Check out some examples by Twitter and GitHub if you need inspiration.

4. Sparse fieldsets / Field filtering

Being able to reduce the information a client receives from the server is a subject that does not often come up until the response bloat is already considerable. Maybe your response objects have scaled up to hundreds of fields, or maybe they have few, but one of them holds hundreds of kilobytes of text or HTML. In such cases, limiting the size of your responses can be achieved by splitting the data across different endpoints, but often that is not practical due to existing clients or other reasons.

A different and sometimes better approach is to provide a common mechanism for filtering your response fields for all endpoints. Your clients’ requests could specify which fields they want you to return in your response, or even which ones they want you to exclude (ideally you should provide both features).

This concept of partial responses is not new; however, it’s very useful for large APIs or clients with slow or intermittent connections (such as mobile devices). Newer alternatives to REST, such as GraphQL lean heavily on this idea of getting only what you need from your API (GraphQL takes it many steps further).

Here’s an approach I’ve always liked:

You have a fields request parameter that allows you to specify the names of all the fields you want to return, using dots to indicate nesting. If the fields you want to exclude are fewer than the ones you need to include, you can specify those by using a minus sign in front of their names.

Here’s a simple example:

Let’s say we have the GET /articles endpoint which returns a list of newspaper articles.

Each article contains metadata such as its title, author, date etc. It also contains a synopsis and the actual article text in HTML inside the field huge_html_content.

When our Newspaper client app needs to show the list of all the latest articles, it doesn’t really need to include the entire text for each article. Just its metadata and synopsis should be enough. In that case, all we have to do is exclude the huge_html_content field from the response using the following request.

GET /articles?fields=-huge_html_content

Alternatively we could accomplish the same thing by requesting all fields without the huge_html_content field, but that would make for a longer request:

GET /articles?fields=title,author,date,synopsis

Let’s assume that the author field contains a nested object with fields such as the article author’s name, email and bio. Since we will only be showing a list of articles, all we need is the author’s name. The rest of the author’s fields are irrelevant.

In that case, the above request becomes:

GET /articles?fields=author.name,-huge_html_content

Notice that we can include and exclude fields per nesting level, so the above specifies that we want all fields in the root object except huge_html_content, and only the name field inside the nested author object.

5. Advanced pagination

The most common approach to reduce the amount of data a client receives is pagination. The usual combination of offset / limit parameters has been around for decades in SQL and the web has adopted it almost unchanged. Chances are that you will need it a lot in your API so make sure to implement it right.

The Link header

The link header is one of those web technologies that is specified in a very abstract and academic way in its RFC but in practice, you don’t see it used that often. When used in the spirit of HATEOAS (see below) it should boost the discoverability of your API by your clients, bring it closer to self-documentation and help make versioning a thing of the past.

If the above sounds too abstract for you, what the Link header basically does; is provide links to retrieve the rest of the data which was not returned by the current (paginated) request. It could provide links to the next and previous pages of data and even add links to the first and last pages, the way GitHub does it.

Rapidly-growing collections

When you need to combine pagination with collections of data that constantly receive new entries, the usual limit / offset pagination parameters will prove inefficient. Every time a new entry is added to the beginning of the collection, the offset that was previously sent to the client is invalidated, since it will have moved by one place. Furthermore, the client has no way of knowing how many new entries were added between two requests.

A better approach to keep clients in sync — while at the same time limiting response size for each request via pagination — is to use the unique ids of either the newest or the oldest record available in the client. This approach works only when unique ids are available for each record and combines the limit parameter with before_id and after_id parameters instead of the usual offset parameter.

Therefore the client can use:

• EITHER the id of the oldest entry it knows (last entry on the last response) as the before_id request parameter, when it needs the next page of (older) entries (created before the oldest entry the client knows).
• OR the id of the newest entry it knows (first entry on the first response) as the after_id request parameter, to retrieve a page of new entries that were created after the newest entry it knows.

6. Accept-Language header

This very important and ubiquitous request header tends to be overlooked quite often although it is very useful. What it basically does is provide a list of languages the client supports (in order of preference) to the server.

I’ve seen many APIs using a custom parameter for language selection and ignoring this header, although most clients send the header by default (respecting the end-user’s localization preferences). You can use this when your API needs to provide any kind of localized content.

THE END

Thank you for reading this far. I hope you haven’t found this series of posts too strenuous and that they have provided at least a tiny amount of relevant information to your quest of building a great API.