REST API Development tips and best practices — Part 1
Introduction and planning
This is the first in a series of three posts on REST APIs.
• Part 1: Introduction and planning
• Part 2: Schema suggestions, common mistakes and deprecation
• Part 3: Documentation tips and moving beyond the basics
INTRODUCTION
REST APIs are everywhere these days. If you are developing network-enabled applications, chances are that you have consumed at least one in your code, and quite possibly you are reading this because you are about to write one of your own.
What this series is and what it isn’t
This series of posts is not an introduction to REST. It assumes that you have a clear understanding of what REST is and what it aims to accomplish. It’s also by no means a comprehensive Best Practices article on Web API design and development. There are plenty of those already available online, and I would suggest that you read as many as you can.
What it mostly is, is a list of very common and often overlooked mistakes people make when designing and implementing REST APIs. I know, not only because I have encountered them in other people’s APIs, but because I’ve made some of those myself. :)
Why this was written
In a perfect world, the first thing anyone should do right after developing a REST API, is write a client application for it. I’m not talking about a UI with a list of endpoints that people can call, rather than an actual use case of your API. There’s nothing more eye-opening to reveal the inefficiencies of your creation, than actually having to use it properly yourself. I know this, because I’ve had to do it dozens of times.
Our world is not perfect of course, and most of the time a back-end developer has to move on to other things after they’re finished with the API implementation, but only until the API consumers have to summon him back to atone for his sins. I’ve been in that unhappy position many times as well, so hopefully my mistakes and tips will help you save time — so that you won’t have to come back to your API code until it’s time to add new features.
PLANNING
1. Before you start
Study as many REST API design and development documents and books as you can find! I won’t be mentioning common tips here, such as using noun instead of verb URIs (oops, I just did…) etc., but that’s because I assume you’ve already read about them elsewhere.
Consider alternatives
REST is quite popular as a paradigm, but it comes in many flavors. Before proceeding any further, you should take a look at as many of them as you can. While you are at it, don’t forget to check GraphQL — an entirely different approach, which not only seems great for certain use cases, but is also used by Facebook (who developed it) and GitHub’s new v4 API.
See what others have done
Check out and actually try to use as many popular APIs, from reputable organisations, as you can. Here are a few to get you started: GitHub, Stripe, Twitter, PayPal
2. The Foundation
Don’t assume your API will stay small — it almost never happens. Even if you’re the only one using it for your own client app, pretty soon you will need an extra endpoint, pagination, analytics, a test mode and who knows what else, and that’s even before your app becomes popular and other people start requesting access to your API…
OK — maybe that’s a stretch; but as it usually happens in software development, if you don’t lay a solid foundation early on, you will pay for it dearly in the long run. Since APIs need to be consistent to be easily consumable, trying to add features to them using duct-tape won’t only make things harder for you, but for your clients as well.
Furthermore, the better you get at keeping things clean, the less time it takes to complete them and pretty soon…
3. Specification
Talking about a strong foundation; try to create the specification for your API before actually starting to write code. I know it sounds boring if you haven’t done it before, but it pays off in the long run and saves you a lot of time and hassle.
OpenAPI Specification (OAS — formerly Swagger) seems to be the most popular way to go at the moment, as more and more organisations and alternatives are converging on it. Alternatively, Postman is a great API development environment but it’s only free for individuals or small teams of developers.
I would strongly suggest not to ignore this tip, unless you are developing a small internal API that only you will be consuming.
4. Versioning
I won’t focus on different versioning approaches in this post, as there are many dedicated resources about this subject online. I will only try to highlight how important it is that you pick a robust versioning approach and follow it religiously. I will also present an approach that has served me well over the years.
Why it’s important
A strong versioning approach provides peace of mind to your consumers (and therefore your end users) because it prevents API changes from disturbing the functionality of existing client applications. Even when client applications themselves are developed by the same developer or the same organization as the API, there are many scenarios where deprecated clients might still be in use at the time an API update is available.
A very common occurrence of this phenomenon is mobile or desktop OS client applications, where the developers are rarely in full control of the update cycle. No matter how good and automated your client update mechanisms are, you can almost never be 100% sure that your users are running the latest version of your client apps.
iOS has a very reliable auto-update mechanism for applications enabled by default, yet depending on a user’s preferences and network status, their device might not have had a chance to update your client app the next time it’s run. The same is true for newer versions of Android, while older versions are even more likely to lag far behind in app updates.
Desktop apps, on the other hand, might or might not provide an auto-update mechanism — but those are most of the time opt-in and there are always cases (such as enterprise environments) which do not allow for immediate updates. Even browser client apps might (and in most cases should) be decoupled from their backend and therefore have an entirely different release cycle than the API.
Communicate changes to your consumers
Even if you have the perfect API with the best versioning system in place, it all means nothing if your consumers are not aware of its full potential. Communicating changes and new features is crucial for your users to be able to take full advantage of them. Even more important is warning your users early enough about aspects that are going to be deprecated or removed, so they have enough time to update, test and ship new versions of their clients.
Keeping an up-to-date documentation and releasing detailed changelogs for each version is of utmost importance and should be part of your development cycle.
Major / minor version approach
A versioning approach that I’ve come to appreciate a lot over the years is a major / minor version scheme. Your API is characterized by a major version number, which (if you do things right) is updated only when there are breaking changes, and a minor version number, which is updated anytime there are small incremental additions.
Viewed from another angle, each major version guarantees that the structure of the API resources and schema won’t break for the duration of its life cycle, no matter how many minor versions are released in the meantime. Minor versions are backwards compatible.
It’s common for the major version to be part of the base URL of the API (e.g. https://api.myserver.com/v1/endpoint) and the minor version to be selectable via a request header. When the request header is not provided, the latest minor version is used. There are many popular APIs that follow this approach, with my favourite one being Stripe (which uses release dates for minor versions).
While I will be mentioning this approach again when talking about updates to your API in the following sections, it’s up to you to pick your own formal versioning system that works best for your requirements. Just remember that once you do, you will have to commit to it.
5. Testing
As an integral part of the software development process in general, testing is essential for your API. Since APIs usually serve as the link between the single-source-of-truth that is the server and one or more client applications, they have to be reliable under all circumstances and cannot afford to go into production broken in any way. Moreover, since they are usually exposed to the web and therefore vulnerable to a large number of risks, they have to be tested thoroughly for a large number of common and not-so-common scenarios.
Manual testing by calling your endpoints via tools like Swagger, Postman or REST console, can only help you during the first baby steps of development, when you are still testing the various foundational features. Maintaining a suite of automated tests from early on makes a world of difference to the quality of the finished product.
Due to their nature, APIs are ideal candidates for functional testing. Unit testing their internal implementation is very important (like it is for any other piece of software) but, in my opinion, functional testing an API and likening it to a black box provides much more value for the time you will spend on it.
Maintaining a test database that you can set up and tear down while testing is an essential part of the process. Keep adding to it data that have contributed to past issues (observed during development or reported by your consumers) and enrich your test suite with regression tests to make sure that you got rid of those bugs once and for all.
Don’t ever assume that your validation of input parameters is complete until you have tested even the weirdest of edge cases (don’t forget that your client apps could be quite buggy themselves). Finally, invest time in a proper logging infrastructure to catch those few remaining runtime bugs, observe consumer behaviour and use that information to create even more testing scenarios.
6. Deployment
As you might have already noticed (or will notice as you keep reading), this post focuses heavily on how to create an API that does not break things. The most terrifying phase to break things in almost all software products is, of course, deployment.
In most cases, if you’ve done testing and versioning correctly, chances of encountering a problem during deployment are very low; however, here are a couple of tips that might be of extra help.
Decouple everything
Here’s a very common scenario:
Your API provides data to a couple of B2B clients developed by external organizations, serves a couple of mobile apps and a web app developed by your own organization, and is an integral part of a server application — which also happens to be accessible through the web, for administration purposes.
That may sound complicated, but I’ve seen it many times in the past in simpler or more complex variations. The key point here is that the API, although being the cornerstone of an organization, is often tightly coupled with at least one server app which has other responsibilities.
That means that when it’s time to update the server app, if something breaks, there is a good chance the API will break as well. If that happens, the API will bring down all its consumers with it, and eventually the end users’ wrath will descend upon the company.
Another deficiency of tight coupling in this context is that the release cycle of one app affects all the others. That will require complex planning and orchestration of deployments. Furthermore, it makes no sense from a business point of view (each app should be released in its own good time — without affecting the rest).
If your API and the rest of your apps are developed as independent modules of your infrastructure, then all of this goes away. Someone might argue that having more separate modules means more points of failure, but that can be solved with proper versioning and testing. Furthermore, even if the new version of your API does break upon deployment, your team will be the first one to know. In this case, if nothing better can be done, you can always delay your clients’ releases for a couple of days until you are sure that everything works properly.
One dangerous case that you have to guard against with rigorous testing, is when the new version of your server app breaks the existing version of your API. Again, the only way to avoid this is (of course) through a strict development process with proper versioning and testing. Treating all the parts of your infrastructure as distinct modules encourages that approach even further.
7. (Graceful) Deprecation
At some point in time, as your app matures, it is inevitable that you will have to deprecate your current major version of the API in favor of a new one. Client apps that are still under development will have to upgrade to the new version as well, but that might take time.
Moreover, there are cases when even immediate action from the developers does not mean immediate updates to the client applications. While updated versions of web apps can be made available to the end users as soon as they are released, older versions of mobile and native OS apps tend to stay around for a long time. Some users don’t really understand (or are not even aware of) the update process of apps on their phone or their computer. If those apps suddenly start malfunctioning, the users’ response might be to just stop using them.
Deprecation of a major API version should not mean interruption of service. The deprecated version should continue working exactly as it did, for a period of time (as long as possible) that will be communicated clearly and repeatedly (as it nears its end) to your consumers. Ideally, older versions of your API should never stop working, unless they pose a security risk, or are otherwise causing harm to your backend.
8. Often-ignored good practices
You can use plural or singular forms for resources, but not both
/cars is fine and so is /car, but you can’t use both. Choose one form and stick to it throughout your API. If you don’t, you will be confusing your consumers and maybe even your own developers or yourself in the future.
Return updated objects in PUT and PATCH responses
After a successful request via PUT or PATCH verbs, the client usually needs to know the state of the updated resource. Since it’s quite possible that multiple clients might be updating it at the same time, this is the only way to keep each one informed about the state of the resource at the moment the update took place.
Cookies and PHPSESSIONID
This one is very common in PHP-developed applications. The API uses its own custom authentication token, but unbeknownst to the developer who uses PHP’s default session handling, all responses also include the dreaded PHPSESSIONID cookie! This causes all sorts of bugs in the long run, because no one (including the developer) is informed about the cookie, because no one thinks they put it there.
Cookies, in general, are not explicitly prohibited by REST. They can cause trouble if used incorrectly (for example, if you try to access a separate domain that should also be authorized via your API mechanism), but they are not banned by any specification. What should be avoided, though, is using more than one forms of authentication at the same time.
Regarding PHP’s session handling, you should never rely on the default PHP cookie approach for your API, unless you want the PHPSESSIONID cookie to act as your API’s authentication token (which I don’t recommend in any way!). My advice would be to implement a custom session-handling mechanism that implements your preferred authentication method.
Don’t expose internal errors or implementation details
Over the years I’ve seen this on countless occasions, even in popular APIs by big companies. It’s not cool to expose internal (such as SQL) errors in your responses! Find a reliable way to log those for future reference, but translate them to a generic 500 — Internal Server Error in your API responses.
This is one of the first things you should do early in the development process. Security should be a paramount concern for a service that is exposed on the web.
This was the first in a series of three posts on REST APIs.
You can read the next one here:
Part 2: Schema suggestions, common mistakes and deprecation
