Real-time subscriptions are essential when building a successful event-sourced system. They support the following sequence of operations:
- The domain model emits a new event
- The event is stored in the event store
- The command is now handled, and the transaction is complete
- All the read models get the new event from the event store and project it if necessary
Most of the time, subscriptions are used for two purposes:
- Deliver events to reporting models
- Emit integration events
Subscriptions can subscribe from any position of a stream, but normally you’d subscribe from the beginning of the stream, which allows you to process all the historical events. For integration purposes, however, you would usually subscribe from now, as emitting historical events to other systems might produce undesired side effects.
One subscription can serve multiple event handlers. Normally, you would identify a group of event handlers, which need to be triggered together, and group them within one subscription. This way you avoid situations when, for example, two real models get updated at different speeds and show confusing information if those read models are shown on the same screen.
Example (real-life horror story)
Imagine an event stream, which contains positions of a car as well as the car state changes, like
Idle. The system also has two independent subscriptions serve two read model projections - one is the last know car location, and the other one is the car state. Those subscriptions will, with great probability, come out of sync, simply because position events are much more frequent than state updates. When using both of those read models on the map, you easily get a situation when the car pointer is moving, whilst the car is shown as parked.
By combining those two projections in one subscription, they could be both behind real-time with updates, but the user experience will be much better as it will never be confusing. We could also say that those two projections belong to the same projections group.
Subscriptions need to maintain their own checkpoints, so when the service, which hosts a subscription, restarts, it will start receiving events from the last known position in the stream.
Most often, you’d want to subscribe to the global event stream, so you can build read models, which compose information from different aggregates.
In Eventuous, subscriptions are specific to event store implementation. We currently provide subscriptions for EventStoreDB as the event database, and also for brokers like RabbitMQ and Google PubSub, which can be used for integration purposes.
The wrong way
One of the common mistakes people make when building an event-sourced application is using an event store which is not capable of handling real-time subscriptions. It forces developers to engage some sort of message bus to deliver new events to subscribers. There are quite a few issues with that approach, but the most obvious one is a two-phase commit.
When using two distinct pieces of infrastructure in one transaction, you risk one of those operations failing. Let’s use the following example code, which is very common:
If the second operation fails, the command side of the application would remain consistent. However, any read models, which project those events, will not be updated. So, essentially, the reporting model will become inconsistent with the transactional model. The worst part is that the reporting model will never recover from the failure.
As mentioned, there are multiple issues with using a message bus as transport to deliver events to reporting models, but we won’t be covering them on this page.
The easiest way to solve the issue is to use a database which supports real-time subscriptions to event streams out of the box. That’s why we use EventStoreDB as the primary event store implementation.
The right way
It is not hard to avoid the two-phase commits and ensure to publish domain events reliably if you use a proper event store. The aim here would be to achieve the following architecture:
Why is this flow “consistent”? It’s because the command handling process always finishes with an append operation towards the event store (unless the command fails, or is ignored). There’s no explicit
Produce call that happens after the event is persisted. A proper event store should have the capability to subscribe to it for getting all the historical events from a given position in the event stream, and then all the new events when they appear in the store. As the
Append operation of the event store is transactional, such a subscription will get the event only once if the subscription is capable to acknowledge that the event has been processed.
In most cases, we want to process events in subscriptions in the same order as they were appended to the store. That’s yet another requirement for the event store infrastructure, and that’s exactly what many message brokers cannot guarantee. The risk here is that when you, for example, project events to another database as a materialized view (read model), and you do it out of order, you can get an obsolete state update executed after the latest one. In that case, the derived read model state will become invalid.
Subscriptions might have different purposes. Most often, you would use a subscription to project domain events to your query models (aka read models in CQRS terms). You can also use subscriptions to transform and publish integration events. I will write more about those use cases another time.
You might also want to read my article about CQRS and projections in event-sourced systems in the Event Store blog.
Originally published at https://eventuous.dev.