Sep 3, 2018 · 2 min read
Hi Jesper,
We’re using Kafka for about 2 years already for similar kind of apps (event-sourced/CQRS apps) and it works pretty nice.
Let me share some of our experiences on how to address problems raised by you :
- Loading current state: in our approach we keep current state cached in another State topic, so we don’t need to rebuild it each time and can leverage from accessing latest persistent state that is always strictly consistent with Event stream. In some (very rare) case when State needs to be rebuilt, on can run an instance (can be a shadow) of the application that will rebuild new State based.
- Consistent writes: I believe there is some confusion here, because that is inherent feature of Kafka, I mean the way how Kafka consumers work allow to implement single-writer pattern natively and without any downsides raised in your post. In our design, we’re using flow like this: 1) Command Processor that receives a Command (from Command Topic) on a single thread (no other processor can process this partition at the same time — guaranteed by Kafka) and applies that Command on the current State (which is also consistent with latest Event) and emits one or few Events; 2) these Events are persisted on the Event topic and also forwarded (in memory) to downstream Event Processor, which applies Events on the current State and updates state in the Store (which is effectively topic). 3) After that command processing transaction is committed and Command Processor can proceed to the next Command (if it’s available). All processing is quite fast (depends on the business logic normally under milliseconds or fraction of it).
- Overall, Kafka and Kafka Streams are quite good fit for implementing event sourcing system, at least in our experience. Of course, they cannot be used OOTB and require additional effort to implement the layer of abstraction ( that models Events, States, Aggregates etc ) on the top of it, but it’s a thin layer and once completed and it’s quite easy to use. Another very useful feature of Kafka is built-in AVRO support which allows to guarantee consistency b/w services.
WBR,
Sergey