MCJSS: An Architecture for modern APIs
2016 has come, leave your LAMP apps once and for all. Start making robust, tested and scalable apps in a short period of development.
Before the journey starts
I wrote all the API code examples in Ruby (2.3.0) and with some Rails magic, but I bet you are smart enough to extract the concept and ideas regardless the language used.
MCJSS: Model Controller Job Service Serializer
Models must be used as data structures. It’s nice to use an ORM, they save lots of time. Models will contain code about relationship with other nested/embedded models and sort of the representation logic about the data, but never, I repeat, NEVER, business logic code.
Writing business logic code in your models will take you to maintain big and fast growing models, making that a daunting task.
It creates problem when refactoring, when iterating over your business model, and for sure, it’s painful to test business logic embedded in a model. Having to mutate the data of the model instanced to get the desired state and that being done a cuple of times for each test case, that drives you to unclear and unmaintainable tests, breaking them each time you change your data structure or you add more complexity on it.
Remember, models as data structures, no business logic inside.
I personally tend to use PostgreSQL with ActiveRecord because I find really comfortable to delegate the database tasks into Amazon RDS so I can focus developing and not doing systems. But seriously, use the database and ORM that fit better your business needs and your usage, avoid fanatism, be professional.
Controllers must be used as simple classes working as glue, enrouting our requests to a service and rendering a response throught a serializer.
Requests with need of a response body from the API
For example if we write a registry endpoint, as an API user I expect output from the API with response about the operation performed
Requests with no need of a response
We use controllers here as the entrypoint for a job, enqueuing the job and returning 202 status code, standing for an accepted request.
With this simple controllers your tests can be done in a few lines without having to test business logic inside the controller tests. Isn’t that cool?
For creating REST APIs in Rails I personally like Rails API, it’s really easy to integrate and also, they’ve been merged into Rails 5, so the will become the Rails standard for API Controllers.
Jobs are simple, without any operational code, and they act as wrappers for a service operation.
Running process inside jobs you waranty yourself not getting overloaded for multiple process running at the sametime and taking your Memory/CPU usage to the limit. This making you resist intensive period of requests and delegating the charge in the scalable workers force. Being able to enqueue tasks even if you platform isn’t able to process more task at that moment. Fire and forget.
That being said, jobs must support retrying, assuring with that your backend is ready to recover from failed tasks or exceptions inside a job. The job queue manager will try running the task later following it’s retry policy. The more tasks you do inside jobs, the more you will be able to reduce throwing exceptions to your API clients.
Your jobs are working as wrappers for services, so testing them doesn’t take you further, personally, I write no tests for my jobs.
My choice for Rails apps as the job engine is Sidekiq, it works great, it has a nice admin panel, integrates with ActiveJob and runs on the top of Redis, the concurrency is adjustable from environment variables, it also support job scheduling.
Services must be designed for idempotency, that meaning you can execute the same task multiple times without affecting to the state of the system, even if the task raises an exception, its a good practice to run the service inside a database transaction, asserting that if something goes wrong, the database changes won’t be performed because the transaction didn’t complete.
A service owns business responsabilities, interacting directly with other services for combined operations, third party APIs, and read/write into database using the models.
One service must have one responsability, using another services as dependencies for combined operations.
It’s important to inject dependencies into the service so you can mock them when you are testing, allowing you to write unit tests that tests your real target, not your dependencies.
Example service
As described before, we’ve mounted a POST /webhooks/company_market_updates endpoint where we will enqueue a job that will call CompanyMarketUpdatesService#update with the sent parameters.
Running this service inside the job will make the company using our webhook endpoint able to provide us with information without being aware of failures, because we will receive the request and enqueue a job with its parameters, that allowing us to retry the job if we have any problem, making our system robust and causing no requests/information losses.
A service using other services after it’s process might be enqueuing a job after so we assert that our first part of the job is completed and it’s not.
Above we have a service processing a business operation, after that, the service enqueues a NotificationJob instead of calling the service directly. By scheduling this non-procedural tasks in another jobs, we avoid running its code inside our current job, assuring that it fails, we are not going to retry it from the beginning and it’s not going to affect our job operations. Again, fire and forget. This might only be applyed for some services, sometimes you need to use another service as a dependency for your process, waiting for it’s output to perform its duties.
Service test example
Serializers, they are in charge of the data presentation. They convert the information we had in our models into a, normally we will be using JSON, response, avoiding with this the typical #as_json method call with infinite include options calls spread arround all the different controllers or the #as_json method override with the included objects inside each model.
Serializers will allow us having nested objects, serialization properties and having nested collections serialized, making our response code clean.
Also, serialization frameworks might be built on the top of ORM magic, so you might decide choosing a ORM suitable for a serializer library.
In Rails, ActiveModelSerializers works great with ActiveModel inherited ORM, which most of the Rails ORMs are. Also merged into Rails 5
