Messaging in a Service-Oriented Architecture
Service-Oriented Architecture (SOA) has proved its strengths in virtually all industries to produce scalable and evolutive systems. SOA has many meanings, but its core idea of breaking a complex application into smaller, reusable and loosely coupled services has come a long way since the late 1990s. We’ve embraced the concept from day one and after three years, we now have about twenty domain-driven decoupled services structuring our backend applications. It allowed us to use the languages we liked — like Kotlin, Elixir — and the tools we wanted — like Event Sourcing — for the task at hand.
But having multiple services comes with its own challenges, the first one being how to make services communicate with each other? There are a lot of strategies out there, and this article is a return on experience on the ones we use at Memo Bank.
Let’s begin with two services and this simple use case as an example:
REST over HTTPS
Like many backend applications we started with REST calls over HTTPS.
Cyclic dependencies between services need to be avoided at all cost, so contracts can’t directly tell customers to activate the new customer as soon as the contract is signed. Instead, customers has to send requests to contracts regularly to know when the task is finished.
HTTPS is a great start, and definitely something to keep in the toolbox: it’s simple to set up, widely supported, and also used to expose APIs to frontend applications. But there is one big issue with this scheme, it’s synchronous (although not necessarily blocking). It means that while contracts is working, customers is waiting with an active connection. When used at scale, it means that the slowest service can slow down the entire system. It also means that when the contracts service is unavailable, the customers service isn’t either, which is called cascading failure.
How did we tackle this issue? We used two different patterns: discrete and continuous messaging.
Discrete messaging
Let’s take the first use case : customers wants contracts to send a contract.
For this, we can solve the problem using a discrete messaging mechanism: queues. You can see a queue as a high-availability third-party mailbox. Even when the main service is unavailable, the mailbox accepts messages. Multiple instances of the same service can consume the same queue, in which case the queue acts as a load balancer and ensures every message is handled at least once. Once a message has been handled, the mailbox can forget about it.
We use queues to send commands (e.g. “SendContract”) from one service to another. It has several advantages:
- The caller service doesn’t rely on the availability of the receiving service, and can resume its work as soon as the command is in the queue;
- The receiver can handle the commands at its own pace, and we can easily scale the receiver service up or down depending on the load of its queues;
- As a bonus, failure can easily be isolated and handled manually (we hope to cover the dead letter queues topic in another article).
Continuous messaging
Now let’s see the second use case, when the contract is signed, contracts needs to let customers know it happened so it can activate the customer.
It’s tempting to use another queue here, but as we said before we don’t want a cyclic dependency, so contracts doesn’t know anything about customers. Thus, it can’t send an “ActivateCustomer” command, and it can’t know when or where to send that command.
We solved this problem using a continuous messaging mechanism: streams. You can see a stream as an ordered sequence of events that can be consumed in real time but are also made available over time. Contrary to queues that are unrelated ephemeral commands, streams tell a persisting story.
Each service at Memo Bank broadcasts a stream of events describing the lifecycle of its resources. Building and maintaining these events is an integral part of any development, whether or not this event is immediately needed. It’s part of the routine, just like automated tests and metrics.
These events thus become a reliable timestamped log of immutable facts. And since they are part of the API of the emitting service, they can be consumed by any other service, both in real time (one by one) and in the future. These events have an immense value for traceability and data analysis.
To sum up:
- Queues are used to send commands to a specific service;
- Streams are used to expose facts from a specific service.
All about dependencies
The diagram above might make you wonder though looking at the arrows, doesn’t it introduce a cyclic dependency between customers and contracts?
No, it does not. A dependency is not defined by the direction of the data, but by the knowledge services have of one another. Here, customers knows about contracts, it tells it what to do and it listens to its story. But contracts knows nothing about customers, it doesn’t need to know who is sending the commands, nor who is listening to its story.
Both the queue and the stream are part of contracts API, and customers depend on this API.
Having a naming convention for commands and facts is very important to convey this idea. We always use a base form for commands, like “SendContract”, and a past participle form for facts, like “ContractSent”.
Note that it conveniently matches with our Core Banking System architecture based on CQRS/ES. In this terminology, commands are the same, and events are facts.
How to choose the direction of the dependency
Given the principles explained before, this solution would be just as valid:
But if both solutions are valid, how to choose one over the other? Well, it’s up to you.
It all comes down to the direction of the dependency you want to set.
Here are some questions we usually ask ourselves :
- Can one service easily be agnostic of others?
Here for example, as long as we give the content to contracts, contracts can be totally agnostic of which service is using it. It’s harder to imagine customers being agnostic about the fact it requires a contract. That’s in favor of A. - What if one service was a third party?
For instance, it would not make sense for Memo Bank to outsource customers but it could for contracts. Thus, this is in favor of A as well. - Is one service orchestrating other services?
Implicit orchestration is bad, you can find more about it in this talk by Bernd Ruecker. Creating a customer is a complex workflow which involves many services (sending emails, notifications, creating a bank account, etc.), so customers is probably an orchestrator here. Making the orchestrator depend on other services — and not the other way around — makes the code a lot easier to understand, because the full workflow can be found in a single place. That’s also in favor of A. - Does it create a cycle in the overall architecture?
Even if there’s no link between the two services, they both depend on other services. Let’s say customers depends on users, and users depends on contracts already. If we chose solution B, it would create a cycle with the three services. That’s also in favor of A.
Conclusion
Messaging is one of the first questions we need to answer when creating a Service-Oriented Architecture. Using HTTPS and REST seems like the most straight-forward solution at first, but it has its limitations. We completed our arsenal with queues and streams, and we set mainly two guidelines.
First, every service should stream events when facts happen within this service, even if we don’t need these events yet. These events should tell a story of what happens in the service, like “ContractSent”, “ContractSigned”. This is great for traceability — which is required as a bank — , but also to consolidate each service’s API, and to make the system easier to work with for all teams.
Second, it’s all about the dependencies. Dependencies shape the system, and cyclic dependencies are the number one enemy. Once the dependencies are properly set, the messaging tools are just here to let the data flow in any direction.
Originally published at https://memo.bank/en/magazine.
