The FIFO fallacy of event-driven architecture
Lately, I’ve been asked the same question by different teams:
What do we do about the order of domain events? Our message bus has this nice FIFO feature, but somehow our solution is getting very complex!
Me: If you rely on the order of events you are doing it wrong. It’s an anti-pattern in event-driven architecture.
Them: But how can this be an anti-pattern, our message bus can be configured to guarantee the order of messages in a queue.
Me: Well, just because you can, it doesn’t mean you should.
Coupled services
Event-driven architecture describes an asynchronous messaging system between decoupled services. That means that both the producer and the consumer shouldn’t know or make any assumptions about each other. But ordering messages and relying on them does exactly the opposite.
Our services are coupled by introducing the assumption of the producer that there is a consumer that will process the events in order and the assumption of the consumer that the producer will send those messages strictly in order.
Distributed tightly coupled services lead eventually to a so-called distributed monolith where we get the worst of both worlds: the complexities of a distributed system and the tight coupling of a monolithic system.
Too much information in events
One reason for the ordering requirement that I can think of is that the messages include data payloads that will be processed by the consumer directly. But this is more similar to a distributed transaction than to anything else. Let’s look at a simple “counter” example.
Imagine we have a counter and a service A that updates the counter and publishes an event “counter-updated”. Further, we have a service B which has its own copy of the counter and subscribes to the “counter-updated” event.
Now imagine that service A always includes the current value into the “counter-updated” event and service B just takes this value to update its own copy. Sooner or later we will run into a well-known concurrency issue found in databases: the lost update problem. If the sequence
- “counter-updated: value=1”
- “counter-updated: value=2”
gets out of order, then the counter of service B will be 1 instead of 2 at the end.
The solution is simple: instead of including the real value in the message, service A should only include a reference where service B or any other consumer can find the current value. Now the order of messages doesn’t matter since service B will always get the latest value from the pointed location.
What if instead of a counter the update is about a large data structure. Many messaging systems have a hard limit on the message size. Our new solution just works, because it only includes a reference to the real data.
Too much responsibility in the infrastructure
Finally, the assumptions about the order of events are driven by certain business rules. For example that a customer is always created before being updated. But this invariant is not enforced in our domain model, instead, it has become the responsibility of the infrastructure. Our domain model is now more brittle and logically dependent on the infrastructure, which is clearly the wrong dependency direction.
In summary, it is not a good idea to rely on the order of domain events. Although there are technical solutions for FIFO queues, they are not a good fit for an event-driven architecture.
