A modern solution to testable and maintainable APIs.
Better Practices
I’m going to propose solutions you’ve likely never heard of and write code you’ve probably never seen. You may feel the urge to retch and reach for the “best practices” that make you feel safe. I’d like to encourage you to keep an open mind.
The term “best practices” comes with an implication of time we don’t often consider. It’s better said in full “currently known best practices”. Times change and practices evolve. The practices evolve specifically to solve for the problems of the times.
Despite various publications of results where hand washing reduced mortality to below 1%, Semmelweis’s observations conflicted with the established scientific and medical opinions of the time and his ideas were rejected by the medical community.
- Wikipedia on Ignaz Semmelweis
Ignaz Semmelweis was the first doctor to connect hand washing to reduced infection rates. He evangelized his discovery to anyone who would listen. In return, the community — including his own friends — had him committed to an asylum. He was tortured and eventually died there.
Be Ignaz. Don’t be Ignaz's friends.
The Problem
The industry standard for backend development has, from time to time, made me consider quitting my job, leaving my home, and taking up a quiet quaint life of poverty. I’ve considered making a roomy two-man tent under a shady bridge in San Fransisco my new forever home. I might live in dirt and eat canned food but at least I wouldn’t have to write code that doesn’t scale, isn’t maintainable, and can’t be tested, changed, or understood.
It’s industry standard. These are “best practices”. Billion dollar businesses have been built like this. I’m so afraid of change…
[they respond in unison]
True, true, true, and ok… I can’t argue that the job isn’t currently getting done. I’m arguing that we can do better, a lot better.
Is good, good enough?
Testing
Mocking by creating an object that looks similar to another and passing it as an argument is acceptable. Using black-box tools like jest.mock to hack at a runtime’s import system and hot swap a real module for a fake one is not acceptable. I should not have to mock anything at the module/import level.
No matter the framework, language, or libraries, a testing suite should only require a slim test runner and, if needed, an assertion library. Testing should be easy. It should not require specialized tooling.
Martin Fowler provided a solution for this years ago: the Dependency Inversion principle. Since then, as engineers often do, a simple idea has been taken to the extreme with DI libraries, DI frameworks, injection containers, and dependency registration.
Keep that s*** away from me. I’m want to simplify my code and my tests, not the opposite. Using a DI framework to make testing easier is like trying to put out a house fire by loading it onto a truck, and dumping it into the nearest lake.
Simplicity is the way and there must be a way…
Behavior Isolation
I was once obsessed with the idea of business logic and the difference between it and other logic. What is business logic? Where does it live? How can I find it? If it’s the only thing that truly matters in every code bases why is it so hard to identify and find?
No matter the language, runtime, framework, or libraries every API is really just a lot of the same stuff. Endpoints validate requests, parse request arguments, make some decisions, access some data, and return a response.
The make some decisions part is where all the value is. That’s the business logic — or at least it should be. The problem is most code bases have this logic living intermingled with everything else. It’s difficult to identify or find. If it can’t be identified and found it can’t be protected and valued.
I should be able to open a code base and read the code, at the highest level, and the “business logic” should jump out at me. It should be like so totally obvi. As engineers, we write 1,000 lines of boilerplate to build up to the 10 lines of business logic that creates real value. Those 10 lines we worked so hard for, they should stand out.
Some say the M in MVC is the business logic. Bullsh**. I’ve worked on MVC projects in .Net, Node, Python, and Swift. You and I both know the logic is all over the place. Yes, I’ve read it in a book and I’ve seen it in a thirteen-year-old’s YouTube tutorial, but I haven’t seen MVC isolate and emphasize business logic in the wild.
If a solution requires a team of MIT grads to be used correctly I’m out…
Funnel Inversion
In most projects, there's usually some form of what I call layering. A request enters an application, travels down through the layers, eventually bubbling back to the top as a response. Your layers might be well organized but if logic for a single operation or use case is spread across them, is it easily understood?
We want our code to have a funnel-like shape. As I travel down through the application there should be less code and the code should be less specific. The code that’s important has been brought to the surface to create a broad top layer where the valuable code we care about is in plain view.
As a reader, to understand what an endpoint does in an inverted funnel codebase I have to traverse the code vertically, trying to first understand the child modules and functions before I can understand the top level. I should be able to skim one file to understand the behavior and intent of an endpoint. If I want to know how it works, of course, down I’ll go.
When making changes, the more code that is shared, the harder it is to make a change without unintentionally breaking other things. On the other hand, the more code that's repeated, the harder it is to make a change and be sure all needed updates have been made.
Some engineers worship the great god of DRY. I spit in his face. Contrary to the DRY ideology, there is a time to repeat yourself. I should be able to make a change to a shared piece of logic, but only for a single endpoint — easily and safely. This will require some repetition but with individual configurability.
The Only Escape is Death
As an engineer with a certain level of experience, you’ll come to know that there is no such thing as a correct decision. You can be right in two dimensions but in the third dimension — time — you’ll always, eventually, and inevitably be wrong. Because of this hard truth, I should be able to change my mind about the underlying runtime or framework without having to change my code.
I’ve worked on a dozen projects where we dreamed about changing the framework or infrastructure. Decision makers were always excited when we told them we could save thousands of dollars, 2x our velocity, and fix our worst bugs by switching from EC2/Express to Lambda (just an example). Understandably, all enthusiasm was lost when we told them we needed four engineers dedicated full time for three months to make the switch.
This is the nail in the coffin for the current “best practices”. Solving this problem is the differentiating factor for the Abstract & Compose Design Pattern. How much money is lost to startups and enterprises alike because they can’t make the changes they need to?
I should be able to write my code agnostic of the framework. I want to deploy to AWS Lambda, change my mind, deploy to AWS ECS, change my mind, deploy to Vercel, change my mind, deploy to GCP Cloud Run, and so on. Not because I’m manic, but because that's real life. Sadly, the industry has convinced us it’s not possible so don’t waste time talking about it.
It is possible, it has to be…
The Solution
To best understand the why and how we’ll take a snippet of common realistic API code and go through the steps to create a solution that solves for our stated problems.
A Quick Review
The problems we want to solve in our demo code:
- Tests should be easy to write. They should not require any module or file system mocking. They should not require a specific import order or a specialized testing library or framework.
- The business logic should be isolated, naturally emphasized, and set apart.
- Logic should be bubbled to the top so that the boring and repeated is pushed below and the important and special is at the surface in plain view.
- The implementation should not be dependent on, or have any knowledge of, the underlying runtime or framework.
- Boring utility code (like query string validation, JSON body validation, cors, etc.) should be modular and usable in any runtime or framework. Just because I move from AWS ECS running Express to AWS Lambda I should not have to change my code.
Genesis
In the beginning, there was an endpoint function with some logic for logging a user in. As an endpoint, its job is to validate the given credentials and return a token. Let’s assume it’s currently running on Express.
We’re skipping a few issues that would be really hard to step-by-step demo around module/logic organization. This endpoint is not layered, all the logic is already at the top. This means, there isn’t a function call like UserService.findOrThrow or AuthService.throwIfAuthMismatch where we would presumably push business logic into.
Step 1: Abstract the Framework
The first thing we need to do is remove the Express-specific (req, res) arguments and res.json function calls. We want a deterministic — or as close as you can get with a networked API — function that accepts the same input and returns a result — regardless of the underlying framework.
Not only will this help remove our strict framework dependency but it will make our code more testable.
We’ll do two things:
- Create a generic Request interface that modules specific to any framework can conform to and all endpoint functions can accept as input.
- Create a function that wraps ours, taking Express-specific arguments, converting them to our new generic interface, and applying the response of our function to the Express response.
Our generic request can be as simple or complicated as we need. If you think about the request object in different frameworks they’re typically very similar. They all have constructs like a body, query args, URL, IP address, host, and headers. As long as it’s not specific to a framework.
This function shape is one that we’ll see often so let’s name it… “hook”. When composing functions, all the functions we compose (aside from the endpoint), will conform to this hook shape.
In our useExpress hook, designed for the Express framework, the FrameworkArgs is (req: Request, res: Response) and the FrameworkResponse is actually void because the response is applied as a side-effect to the response object.
Using our generic Request interface and useExpress hook I’ll refactor the demo function to accept the Request type as an argument and return a result.
Excellent, we now have a login function with all our valuable code and it knows nothing of the framework.
We’re using the compose function from radash to do the composition. The compose function is super simple. If you don’t like Radash or don’t want the additional dependency you can implement it yourself in two lines.
Compose is just syntactic sugar. When you want to pass one function to the next, using compose you can write them in a much more readable series instead. If that doesn’t make sense, take a look at the following snippet.
If you’re familiar with Python this should be screaming decorators to you. They’re functions that take a function as an argument and return a function. If you have more of an OOP background just trust us… you’re in a safe place now, no one can hurt you here.
Dependency Inversion
By abstracting the framework we’ve also improved testability a bit. We can now call our function without needing to mock any global framework objects (like an express application object). However, our function is still referencing modules it wasn’t given as arguments. To test, this would require file-system/module mocking which is a stated problem to solve.
A bit of Dependency Inversion can go a long way here. I’m not going to install a DI framework, I’ll just extend the arguments we pass to our function and create another hook function to do the dependency injection.
Along with the req I’d like to pass services (the dependencies) now. I’m going to wrap them both up into an object I’ll call Props. A bit like React, this is everything our function gets passed. Because we’re doing function composition, it keeps things simpler to work with unary functions only.
Our function now gets an object with a services property that we can pull from. As a rule, I only put things into services that are non-deterministic. For example, the auth module’s compareCreds function is deterministic. Given the same input, it will always produce the same output (as any password comparison should). I won’t add it to services.
The current version of our demo code won’t actually work. We’re stating a Service type, and deconstructing services from the Props but we still need a hook to inject those service dependencies.
Here it is, our hook takes an object with the services you want to be injected into your endpoint function. This is a bit of a naive implementation. A more robust implementation accepts functions or promises as the values to the services argument and then makes sure they are resolved before calling the endpoint function.
If you’re into Abstract & Compose and you want to write your own hooks let me drop a bit of advice. Split your hook into a logic function —
withServicesabove — and a wiring function —useServiceabove. This makes testing your hook much easier!
I hate on DI frameworks but they do some things well. One of those things is ensuring an object gets all the services it depends on. With that said, DI frameworks are still trash because we’ve done that here with a few simple types.
Our endpoint states a Services type that it passes to the generic Props . Because of this, the endpoint can only access items in services that have been declared on the Services type. Our useServices hook also requires the Services type as a generic parameter and uses it to enforce that all attributes on Services are provided as an argument to the function. Typescript is now validating that our function has all the dependencies it needs to run.
Utility Abstraction
At this point, we’ve created a design I’d personally be happy to work on. We can still do better by taking advantage of the new hook concept. We have some utility logic mixed in with our valuable business logic. For the sake of isolating our business logic, removing what is boring and repetitive, and making common tasks more reusable, we’ll add another hook.
My goal is to remove and abstract away
- reading of the request body
- validating the body attributes
- erroring if anything is invalid
Before we dive into the solution I want to point out a few key concepts I’m using as a guide.
- First, the data in the request is unsafe and untrusted until it has been validated and parsed.
- The
reqshould be treated as an immutable property so other hooks can depend on the originalreqalways being accurate. - For the sake of clarity, our function should clearly define the arguments it requires.
Because of these three things I’m going to extend our Props type to include a generic args attribute. When we successfully validate and parse data from the request we’ll place the processed value in args without modifying req. Inside a function, for 99% of cases, you should only access request data through the args .
We’ll update our function to declare the arguments it requires as type Args and pass that type to the generic Props.
Our new hook will take the Args type parameter as well to help ensure that we are always validating all required arguments to a function.
Our new hook allows us to specify the schema and let the hook take care of the rest. The logic specific to the endpoint function stays in the endpoint function while the utility logic of validating and parsing is pushed below.
We can use this hook pattern to solve, in my experience, every common problem I’ve ever faced as an engineer. I’ve written hooks to handle CORS, caching, logging, alerting, metrics, billing, versioning, websocket messages, SQS record parsing, and more.
Bonus Features
There are some features you’ll get from using this pattern that we didn’t explicitly mention above.
Functional Infrastructure
Every endpoint is an entry point. By using compose to chain together the hooks and endpoint function we get a single self-contained function that has everything it needs to execute — accept the framework. This allows us to decouple the code from the infrastructure.
If every endpoint is a function in a file, the infrastructure is limitless. Our endpoints are like micro containers, able to be moved here and there without change. I can move a file that exports an Abstract & Compose endpoint function from AWS Lambda to GCP Cloud Run or Heroku without needing to make changes.
Language Agnostic
Although the initial library was written in Typescript, Abstract & Compose only needs functions to work. From my own experience I know I could implement Python, Swift, PHP, Java, Kotlin, Go, and C#.
An OOP engineer with a few beers and a slow Monday could probably find a way to make Abstract & Compose work via classes as well.
Performance
Endpoints using the Abstract & Compose design pattern tend to be faster than an equivalent endpoint not using the pattern. I’ll preface this by saying the amount of testing done has been minimal. However, it doesn’t take one of SpaceX’s rocket engineers to reason about why a composed function would be faster.
Let’s look at Express again, as an example. When a request enters it originates in the global application logic. That application routes it to the correct handler based on the path and method. As the request travels to the handler it goes through all globally registered middleware. The entire application is involved.
The opposite is true with composed function. Every endpoint function is composed of only the hooks it requires to execute. Assume a large project has 100 hooks and 100 endpoints. They have no effect on a new endpoint. The opposite is true with Express, if you have 100 middleware functions every request must be processed through all of them.
To be fair and maybe more clear, Abstract & Compose does not make your API faster. It isolates the factors that affect a function’s performance to only that function.
Exobase
I created a library, Exobase, that gives you common root hooks, auth hooks, request validation hooks, and the generic Props typing out of the box.
To be clear, as you saw above, no library is needed. The code in the Exobase library is rudimentary at best. You could probably have any junior engineer write up the wiring given good specs. As the creator and maintainer this a feature I hold dear.
Exobase provides a handful of packages you can use:
- The
@exobase/corepackage provides thePropstype, a functional structure for errors, and a utility for parsing results returned from endpoint functions. It’s three files. You could install core only and write your own hooks. See on GitHub. - The
@exobase/hookspackage provides commonly used hooks to do CORS, query validation, json validation, header validation, and dependency inversion. See on GitHub. - The
@exobase/authpackage provides hooks and models useful for wiring up your own authentication. See on GitHub. - The
@exobase/nextpackage provides the NextJS root hookuseNextthat abstracts the NextJS framework. See on GitHub. - The
@exobase/lambdapackage provides the AWS Lambda root hookuseLambdathat abstracts the AWS Lambda framework. See on GitHub. - The
@exobase/expresspackage provides the ExpressJS root hookuseExpressthat abstracts the ExpressJS framework. See on GitHub.
Thanks for reading!
Even the best idea is nothing without a community — as Ignaz Semmelweis found out. If your a fan, please give it a clap, star it on GitHub, and share with a friend. If you have questions or comments I’d love to hear them.
- Ray
