Event Sourcing + CQRS: from theory to AWS — part 2
In the previous article we saw how an Event Store works and what Event Sourcing and CQRS together allow us to achieve.
In this article I’m going to show you a general architecture of Event Sourcing and CQRS applied into a microservice system. You’ll see also some additional benefits of having an event-based system.
CQRS within a single microservice
As we said before we have an Event Store where all the writes will be issued and one or more databases that will be targeted with our queries. How can we keep query databases updated?
There are mainly two ways. The first one is to have a component (we’ll call it Projector or Denormalizer) which keeps quering the Event Store for the last events of each Event Stream. But generally it’s difficult to keep track of all Event Streams inside the Event Store.
The other way is to exploit some publish/subscribe or trigger based functionality given by the Event Store. Most cloud databases offer this functionality (DyanamoDb Streams, Azure CosmosDb Change Feed, Google Firestore Triggers). So implementing an Event Store on top of one of these will give us this possibility.
Once we subscribed to the events published by an Event Store, we just need to process these events in-order and update our query databases.
In a pub/sub mechanism when a message is published if it is not processed or not listened, it will be lost. This means that we need some kind of persistent queue where to store temporarily our events and in a second time process them. The benefit of this is that if our Projector fails while processing an event, that event will be kept in the queue until it’s explicitly removed. Another constraint is that our queue must guarantee at-least-once delivery, otherwise we are still losing events.
As I mentioned, events have to be processed in order. What does it mean? We have multiple streams inside of which our events are ordered. What about cross-stream order? It mainly depends on your business case, but in general if you design your Event Streams appropriately you have to be concerned about ordering only on events coming from the same stream.
As I said, events need only to be processed in order, not necessarily delivered in order (even if it’s recommended). So it will be a task of our Projector to have some order control mechanism.
Please note that every query db has its own Projector with its own queue. This because whenever you want to restart a query db you have to replay all the events from the Event Store. If you use a shared queue you’ll have that:
- If there are 10 query dbs and you restarted only one of them, processing lots of old events will slow down the update rate of all of your query dbs, increasing the consistency delay between your writes and your queries.
- If a query db is in maintenance, you have to reconfigure your Projector in order to deque events when they are processed for all other query dbs.
Using a dedicated queue and a dedicated Projector for each of your query db, you’ll get the following benefits:
- The development of each Projector is isolated
- The restart of a query db is isolated and wouldn’t affect other query dbs update performance.
- You can add/remove any of your query db in anytime without redeploying your Projector and without replaying events for all query dbs.
Great! Up to now we designed architecture of a single microservice using Event Sourcing with CQRS. Can we exploit events in inter-service communication? How can Event Sourcing help us in this?
Distributed transactions between multiple services
You can exploit various patterns for inter-service communication. They are grouped into synchronous communication vs asynchronous. The preferred ones are asynchronous patterns.
Typically asynchronous communication is done by sending a message to a queue which is read by all the interested services. If this message is needed for triggering other writes (which means to start a distributed transaction across multiple services), it’s important that the write and the message publication are done atomically. As we discussed before, the Event Store already do this.
What we need to do is to use some persistent queues in order to avoid losing messages in case of failure while processing them. Lastly we need to ensure in-order processing.
Someone could think of using only one queue as a general message bus. But in my opinion this should be avoided for the following reasons:
- Your entire system relies on your single bus, which is a single point of failure.
- Each of your services may not be interested on the events of all other services. For each service, this leads getting lots of events you are not interested in.
What I prefer is a dedicated bus (or queue) for each service owned by the service itself. This means that each service can choose to what events subscribe (including its own events) and this means increased decoupling.
This includes that whenever you start a brand new service which needs old events from some of your services, you won’t saturate the entire system with events needed by just one service.
In the end our final architecture will be the following:
Keep in mind that each microservice can also subscribe to its own events, if needed.
From the picture you can see that each microservice doesn’t know anything about the architecture of the other. They are only concerned about each others events. This means that if the entire system exploits any message-based asynchronous communication, Event Sourcing + CQRS can be applied in isolation for just one service and we are not creating any other coupling between services.
Conclusion
We saw the general architecture of a microservice system based on Event Sourcing and CQRS.
In the next article I’m going to discuss how to bring all on AWS. This will include the implementation of an Event Store on AWS, order control for event processing and some other additional mechanism for a simple system recovery from occasional failures.
