The dawn of the REST API — Part 2
Beware of backward compatibility
In part 1 of this post, we covered how we work here at Wolox and the different stages of the development process where parallelizing tasks is highly recommended. In part 2 we will review the various drawbacks of having different teams working in parallel and the cost of maintaining a REST API.
Generally speaking, the parallel development process works quite well, however, there is still room for improvement. For instance, sometimes the backend team starts working on endpoints that the mobile team is not using at that given moment. When the mobile teams start working on the screen that requires those endpoints they realize that some changes need to be made to the endpoint to improve performance, load time or user experience in general.
This also works the other way around. For example, the mobile team defined the perfect endpoint specification for the use cases they were working on but then the backend team realizes that in order to provide the JSON and the endpoints the mobile team wants they have to sacrifice performance (by doing extra joins in a table or making the JSON payload too big).
The problem gets worse when you get to the point of having to re-architect part of your application because one endpoint needs to be super fast leading you to start to denormalize tables. This usually translates into a more complicated write or updating patterns were you’ll probably need to trigger background jobs to update all the denormalized entities. This means more error logic and ensuring that in the case a is a server down, background jobs are not lost. If these are not monitored closely they can escalate very quickly and seriously impact your delivery date.
Another headache in the process is backward compatibility of the API. If you haven’t shipped your first public version this problem is not as significant of an issue as there aren’t any users using the application. However, it is still a problem for the different teams. When you have a QA department that is testing different versions of the mobile apps, suddenly features that were supposed to work stop working. This usually happens because the backend team introduces breaking changes publishing them with the staging server while different versions of the mobile apps in the queue for QA depend on the previous version of the staging server.
This problem can be solved by maintaining different versions of the API or by having several staging servers. Note, this can make the development process more tedious and increments development costs by having to allocate more server resources. What ends up happening is either the backend breaking change is rolled back from the staging server or all pull requests for the mobile applications that depend on the old version of the server get updated to support the new API specification.
The moment you made your app public with active users you cannot allow breaking changes of the API to break running client apps. If your user base is still small you can force users to update to the latest version but we recommend deploying a mechanism to do this in advance. If you cannot afford to do this then you must support different versions of your API until the majority of your users have updated to the latest version in which case you can remove the old version.
This is not an easy task. You server logic becomes more complicated because you have to contemplate every running version of the API. You have more code, which means more source of bugs and depending on the life span of API version you have more documentation to maintain. All this translates to more developer hours, less time to introduce new features and more overall costs.
Also, you have to keep in mind that the more software you build the more software you have to maintain. Having your own backend servers comes with maintenance. The amount of time that you expend maintaining those servers will vary on how unique the web service your are building is. Chances are that if you are using Heroku to deploy a small web service that does CRUD operations on top of a Postgres database and has some business logic, then you won’t spend much time on server maintenance. In the case that you have to provision your own hardware (whether it is virtual or bare metal), and you have more complex backend services, you will have to spend more time doing maintenance work.
Either way, sooner or later you will have to optimize the service configuration, check that your process doesn’t have any memory leaks, that your hard disk doesn’t run out of space, that your database has the right indexes configured, and that the majority of the hot queries run on memory. To make sure everything is working as expected you will need to have to configure monitoring services, configure CDN for your static files, monitor the average response times of your endpoints, find out why a particular JSON response is taking too long to be serialized or why your sidekiq process has consumed all the available database connections.
Once your backend starts to grow and you don’t use services like Heroku, which lets you deploy with a git push, that is when then you have to worry about tooling. If you want to iterate faster you have to deploy as fast and as many times as possible. That is what the whole continuous integration thing is about. Deploying a new version of your services should be as simple as committing new changes to your git repository. The same goes for rollback, reverting a change should be as easy as deploying one.
As you can see developing a backend service is not just about coding. There are a lot of operational issues that one must consider. There are times where it is important to have a backend that does a lot of business logic and times where it is not as crucial. If you can do most of the work in the client and make your servers as dumb as possible then scaling your app to handle more users will be simpler. Of course, rich clients have their own drawbacks, like having to code every business logic multiple times when using native mobile technologies or having to go through a review process every time you want to deploy new changes. But there are ways to mitigate or at least minimize those problems.
If your apps make a lot of API calls just to store and retrieve data and sometimes trigger some background processing job like an image or sending emails it may be time to reconsider having your own custom backend service. The good thing about not having a backend service is if your app starts doing really well and you get more traffic day after day than scalability is not going to be an issue for you or at least you don’t have to worry about provisioning more servers or sharding your database.
To add more issues to the mix, when you have more teams, you have more people and more people means more coordination. Which at the end of the day means more time and money. If you are a startup trying to get your first MVP out in the market, chances are you are running on a tight budget. Being able to build the same product with fewer people gives more room to breathe in case you need to pivot.
All these pain points led me to the conclusion that maybe creating a custom REST API to power mobile apps is not the best solution. Don’t get me wrong REST APIs are great, and if you want the community to be able to build other products on top of your service, they are even better.
Great APIs require excellent documentation, clear backward compatibility, and deprecation policies. I never get tired of the reference GitHub API as one of the most well- designed APIs. Like anything else in life, decisions come with trade-offs and there is no rule of thumb or a magical solution but if you are a startup running on a tight budget with a potentially high viral mobile only product then using BaaS may be the best fit for you.
