When ACID went Sour
When monoliths start to buckle under their own weight, they tend to fall apart in smaller pieces. Managing these pieces quickly becomes a new challenge on its own. How big should these pieces be? How do they communicate? How do I ensure they still work in harmony?
Where we had a big stone that was able to cast waves in a pond, we now have some pebbles and sand that create ripples that even sometimes cancel each other out.
One of the biggest issues you’ll start to encounter is consistency over multiple services.
Driven by business capabilities
How do you decide how big these services should be?
There’s a real risk of clustering too many pieces together leading up to new monoliths. Just as there’s a risk of making services too small (table sized) and inherently creating your own Relational DataBase Management System.
A good size would be to isolate functionality based on atomic business capabilities. Each capability can be owned by a Product Owner. Examples would be:
- ability of taking and managing orders
- ability of returning status information from a shipping provider
- ability of pushing messages to users their mobile phones
- ability of knowing things from products
- ability of managing lists of products
This is in line with Domain Driven Development. I won’t go in to all aspects of DDD, but if you are not familiar with it, I recommend on reading some more about that.
Interoperability
Functional implementations usually are crossing more than one of these abilities. For example, when you’d like to show a list of products on your application, you need to consult the service that is responsible for ‘the ability of managing lists of products’ as well as the ‘ability of knowing things from products’. These capabilities can govern several value sets. Think of e.g. an order that contains order-lines. It wouldn’t make much sense to have a service for orderlines and one for order entities. Or a service that contains list entities and a service that contains only relations between these list entities and products.
We need an abstraction level that makes sense. Like ‘product lists’ or ‘orders’. We call these aggregates.
Storage
Since a service becomes owner of a piece of its domain, it should be able to take and guard this ownership. In other words, each service requires its own database, that can only be used by this service and this service alone. And even more-so, the data that is stored should be understandable outside of its context.
Practically, this means that e.g. the productlist contains a list of products, and not only their reference. Also the parts that I need to understand enough about this list, like a product number, a product name and some other attributes. This means a LOT of redundancy over services. But a lot of sensible output as well! And what about keeping all of that in sync?
How’s the headache? Don’t worry, don’t mind the overhead, we’ll come to a solution for this later.
ACID, the drug of RDBMS
ACID compliancy is guarded by your database (RDBMS). We almost blindly rely on it When your application fails, your database is able to perform a rollback that guarantees a level of consistency in your dataset. It also means that we do stuff in a certain order and guard for race conditions.
This is not possible in a Service Oriented Architecture (SOA). There is no — and should not be — one single relational database that contains the structure of the universe. And since ACID is a functional property of your RDBMS, you cannot rely on this beyond your service boundary. It’s nonexistent and it shouldn’t.
Eventual sweetness
But, but, we need consistency! And all this redundancy of information, how can we keep track of all changes? Relax, we’ll come to that in a minute.
Commands
Services that own a particular business capability (like knowing stuff about products, let’s call it service-products) are able to take some commands. Think of CRUD operations or even perhaps some Procedure Calls on specific endpoints of the service. We call these service requests ‘commands’ that get translated to a series of events.
Event sourcing
So when the command gets translated to events, we can apply these to the current state, which mutates the state we have of some object. With event sourcing we use these events as the source of our data, rather than the representation it leads to.
Instead of up-front constructing a data model of an order and keep updating that model and the order, we just keep all events that ever happened to that order (create, update, shipping update, replacement product, whatever operation needed) and replay them. When you replay all these events, you’ll end up with an in-memory representation of that object, in this case the order.
This has major benefits like:
- Inherently construct an accurate audit trail.
We can exactly tell, what happened when by whom, because we need that information to come to the latest order state. Implementing audit trails are a pain in the butt in more conventional systems. They are either too low level or don’t say enough. And it’s usual up to the discretion of the developer to have them at the right spots. - Replay of events
We can now replay events! Need a new service that does some analytics over specific functionalities? We can just replay these events for this service and pre-populate it like it has always been there running in the chain - Immutable History
We store the exact line of events, and don’t allow a mutation on them. Example: in traditional systems we keep mutating an order with several statuses, and therefore have difficulties to find out how the order changed between now and the moment of creation. Since in Event Sourcing the events aren’t allowed to be mutated, we can with certainty state what happened to the order. - Loose Coupling
When a product is updated, we store this event, but ship it to a pub/sub as well. Here other subscribers are able to listen in and mutate their own datasets as well — if needed. These services that subscribe aren’t aware of who created and how to interface. They just know something has happened in which they have interest.
CQRS Storage
Awesome, but that must be really slow, right? Having to rebuild all data constantly from events? Well, kind of true indeed.
When you rely on the state to be determined on events, it would be hard for e.g. your bank to allow a transaction. They first need to stack all changes of the past 30 years to be able to tell if the next change is allowed.
Therefore it might be handy to store a projection of that state.
So what your bank does, is storing the latest version of the derived state. E.g. just the number, but only for read purposes. And these monthly states they send you? They are projections in-between, so if you would like to know the state at a certain moment in time, you can just go back to the closest projection and stack just enough changes on top of that (sequentially of course) to get exactly there where you wanted.
This requires the separation of writes and reads. That’s exactly what this acronym stands for. Command Query Responsibility Segregation.
When you want to know the state of your order, you get the latest state (query), and if you want to update its state, you issue a request to your service for mutation (command).
Our knowledge of what is needed as a model to store data in continuously evolves. It only reaches as far as we can see, but there is often so much more that alters your perspective on things. Because we separate the events that happened to the data from the ability to return or search through it, any structure this data has becomes more trivial. Because yeh, the truth lies in the events, not the representation of them.
Is this bullet silver?
No architectural design is a silver bullet. But these patterns most surely help to
- determine the scope of your services to loosely coupled business capabilities over technical pieces of software.
- ensure audit trails and performant standards
- provide a structure for extensibility with replay functionality
And last but not least, a solution for what the title states, consistency over many services.
