Clean API Architecture 🔵 🟢 🔴
The pattern you need — or probably had but didn’t realize
We started this 6-part series on how to build web APIs by introducing the variety of architectures that have been proposed or put into use by various languages and frameworks over the years. Among the most commonly discussed architectures online is the Clean architecture, which aspires to produce a separation of concerns by subdividing a project into layers. Each layer abides by the Single Responsibility Principle, ensuring each class is only handling one part of the process, and is more easily and thoroughly unit tested.
Applying Clean to Endpoints
The Clean architecture can be used in many domains. In another blog series we describe how we applied Clean to our mobile applications. Today, we are going to talk about how we apply Clean to API endpoints. We call this the Clean API Architecture:
We have the following layers (and colors) which map to the original Clean architecture:
🔵 FRAMEWORKS & CLOUD
↓
🟢 INTERFACE ADAPTERS
↓
🔴 APPLICATION LOGIC
↓
🟠 ENTITY LOGIC
↓
🟡 DATA
None of the layers have visibility into higher layers. They may have references to their child layer, but definitely not their grandchildren
This diagram also depicts our W-shaped execution flow we described in an earlier blog post, this time represented by arrows that start both at the HTTP layer and again at the Interface Adapter layer from a Queued asynchronous job.
Now it’s time to show you the classes of our architecture.
🔵 Frameworks
Any endpoint request must be routed to the appropriate code path through a Load Balancer, Web Server, Application Server, and an API / Web Framework. (We use Sinatra for this last part, but popular frameworks include Rails, Django, Spring Boot, and others). API frameworks offer the most documentation online and is a common (the most common?) architectural structure.
Once a request reaches your API or web framework, however, patterns can diverge widely.
Frameworks like Rails employ an MVC pattern that works well for smaller projects, but have weaknesses when it comes to large, high availability APIs such as ours. Rails models and controllers get fat very quickly when applied to large APIs.
Consequently, everything that comes after the Frameworks + cloud layer is (mostly) novel and inspired by the Clean architecture. We will be making the case for our architecture and our decisions in the remainder of this series.
Supporting classes
In any given layer, we will have one or more supporting classes. These are classes that are going to be single-purpose and have no references to other layers above or below them. They help with code re-use and duplication, and enable us to avoid writing complex god classes in a given layer.
We include cloud services as supporting classes of our Framework layer. AWS services like EC2, SQS, RDS and ElastiCache are supporting classes — NOT “inner” or central layer — because, as others have pointed out, the UI and the database depend on the business rules, but the business rules don’t depend on the UI or database.
🟢 Interface adapters
Once a request comes in via our framework, a Controller orchestrates the processing of the endpoint by invoking a Request object to extract each parameter, validate its syntax, and authenticate the user making the request.
Controllers are the first place our application code is introduced. Controllers instantiate our classes and move data between classes in the 🔴 Application Logic layer.
It’s important to note that Controllers are not the only orchestration object in the Interface adapter layer. We also have Jobs, which are used in our asynchronous queue processing layer (more to come).
Supporting classes
Controllers depend on classes including Validators, which check the syntax of incoming data, and Presenters, which format outgoing data, and Response objects, which map objects and/or hashes into JSON, HAML and other formats.
Socket relay classes communicate state changes to the client over a socket communication channel, such as Websockets.
Request classes are typed data structures that bring together the necessary components for the request being made. This is different from standard HTTP requests, which (assuming CGI) are made up of key/value string pairs.
Response classes are like renderers in Rails, and enable you to return HAML, JSON or other types.
Parameter extractors extract data out of the params hash and converts it to properly typed values, such as ints, floats and strings.
🔴 Application Logic
GET request made to read endpoints are next passed to the Application Logic layer where a Service ensures the validity of the inputs, makes sure the user is authorized to access data, and then retrieves data from the Entity Logic Layer through a Repo (for databases) and/or Adapter (for APIs). Service objects return Result objects as defined by dry-monads.
POST, DELETE and PUT requests made to write endpoints do the same thing as read endpoints, but defer processing by enqueuing Service inputs through our queue — Amazon SQS — and write the data to the Entity Logic Layer through a Job or Service.
Jobs are used to orchestrate side effects, such as sending a socket message through a Relay after a data mutation completes
Supporting classes
In our implementation of the Clean API architecture, the Service class itself assembles a distinct collection of Validator classes to provide an additional layer of semantic validation for a request. Thus, we have two layers of validation — syntactic, happening via the Request layer, and semantic, happening via the Service .
🟠 Entity logic
Entity logic refers to components that are common not only to this endpoint, but others as well. Repositoryclasses, which provide us access to persistent stores like Mysql or Postgres databases, and Adapter classes, which provide us access to APIs, including AWS storage apis like S3, ElastiCache and others. We expect classes in this layer to be used over and over again; classes in the layer above are often single-purpose to an endpoint.
🟡 Data
This is ideally a very simple layer that provides an actual interface into our different storage systems. If you use Rails you will likely be receiving ActiveRecord objects; APIs are ideally returning ruby structs.
Testing
In other domains — such as Android or iOS development — we have created interfaces to our data storage layer. We use dependency injection so that, when we run tests, we are doing things like creating mocked, in-memory versions of SQLite data storage, rather than using real filesystem-backed data storage systems.
On our webserver, because of the dynamic nature of ruby, we use stub_const to overwrite singleton cloud services with mocked versions, or we will point singleton cloud services to local docker containers running Redis, Memcached, Mysql, etc.
Storage systems that power the Data layer, such as Mysql and Postgres, that are implicitly at the bottom of the diagram are very likely going to be process-wide singletons that are ideally injected or mocked. ActiveRecord maintains its connection pool; systems Redis and Memcached will also likely need some kind of global pool or singleton managing access.
Stateless HTTP-based APIs, such as S3, DynamoDb, and others, are typically going to be mocked by instance_doubles or overridden connection parameters that point to local mocks.
🙄 Isn’t this overkill?
We know what you’re thinking. Isn’t this just another example of extra engineering and complexity that I don’t need, and that maybe you don’t need either?
Let’s explore this for a minute or two. Imagine the world’s simplest API — adding a favorite for a profile. This is what it might look like in a simple endpoint:
In fact, this endpoint is implicitly relying on each of the layers we have defined above. Here is how:
The first line, post ‘favorite’ do, is hook into Sinatra 🔵 framework, as you might expect.
The last line is a form of presentation logic that is part of our 🟢 Interface Adapter layer. It is simply a 200 response with no body.
What about the other layers in our Clean Api architecture? Can we see echos of those in this 6-line snippet?
🟢 Request
Params are passed directly without any extraction, and together the two symbols — :target_id and :creator_id — form an implied Request, comprising a target_id and creator_id.
🟢 Controller
Because of the lack of any validation or presentation customizability, there is no need for a controller so there is no equivalent in this code.
🔴 Service
The where clause takes your request — a target_id and creator_id — and finds a domain object — a Favorite.
🟠 Entity Logic
The first_or_create provides the Entity logic — interacting with persistent storage via a special ActiveRecord method
🟡 Data
The Favorite model is equivalent to your Data layer, providing a definition for the object being used.
Each of these steps we outlined in our Clean API Architecture is still happening in our simplified example, they are just happening implicitly. As a result, if we stick to our simple example, it makes things harder to test, and more importantly, harder to decompose as logic grows. How do you add more validations? Modify more classes? Add push or socket notifications? Endpoints become dozens, hundreds of lines long, and thus responsibilities become scattered throughout the code. We also fail to explain what happens if we encounter errors or how we handle authentication.
The takeaway is clear: even the simplest APIs have to make these architectural choices. In the absence of explicit classes and rules, these choices are either assumed, implied or ignored, but they cannot be avoided.
Next up: responsibilities of an endpoint
We’ve hopefully convinced you that this isn’t quite as easy as you might have first thought. Before we show how to build an endpoint, lets spend time rigorously documenting everything we expect our endpoint to do for us, and connect those responsibilities back to our Clean API Architecture.
More in this series
- A visual history of web API design
- Patterns of web API execution flows
- Railway Oriented Programming
- Clean API Architecture ← you are here
- Endpoint Responsibility Checklist
- Code example: Saving a favorite
Other series you might like
Android Activity Lifecycle considered harmful (2021)
Android process death, unexplainable NullPointerExceptions, and the MVVM lifecycle you need right now
Kotlin in Xcode? Swift in Android Studio? (2020)
A series on using Clean + MVVM for consistent architecture on iOS & Android
One more thing…
If you’re a mobile developer who enjoys app architecture, are interested in working for a values-first app company serving the queer community, talk to us! Visit https://www.scruff.com/careers for info about jobs at Perry Street Software.
About the author
Eric Silverberg is CEO of Perry Street Software, publisher of the LGBTQ+ dating apps SCRUFF and Jack’d, with more than 20M members worldwide.
