The Promised Land of Event Sourcing
The guide you should read before starting with Event Sourcing
Every software system starts as the most innocent greenfield project. After all those legacy systems that you have worked on, you believe that this time it’s going to be different. This time it’s finally going to be better. A couple of weeks in and you feel like a new person, you feel proud — it’s your gorgeous baby after all. Suddenly, a few unexpected business requirements later, you realize that it’s no longer your beautiful child. It’s the Kraken that has been released! By you! What have you done? Well… Off you go… To the next project!
If you have ever developed a software system in an inherently complex business domain, I bet you know how complicated it could get. As a software engineer, I had an epiphany about dealing with complex business domains when I heard about DDD (domain-driven design) for the first time. A few years later, working in a financial field, I met some brilliant people who were using a DDD pattern called Event Sourcing. It took me quite a few days to get a grasp of it. It was so different yet so elegant. My only regret now is that I haven’t heard about Event Sourcing before.
I spent quite some time reading articles, books, having discussions with experts in this field, and implementing the pattern myself. As a result, I wrote this guide, which I believe is one of the quickest ways to start with Event Sourcing.
What is Event Sourcing
We’re used to the “classic” way of storing all application state changes as the latest state of our business model aggregates such as an Account or an Order. It’s not unusual to find those kinds of systems to be CRUD-based and built around relational databases. Unlike the “classic” approach, when using Event Sourcing pattern, we persist an application state changes as a sequence of state-changing events — immutable business-related information about things that happened in the past. In a nutshell, Event Sourcing is just a different way to store the data. Let’s take a hypothetical payments related application, for example. Whenever we make a new payment, a new PaymentMade event is appended to the Account aggregate events list and persisted within a single atomic operation. Later, the Account balance can be reconstructed by replaying persisted PaymentMade events.
Event Sourcing allows tracking all business-related things that happened within an application, which is often extremely important in certain domains. I can’t even imagine a bank that would not allow its clients to view their transactions. At the same time, the ability to check what happened to a brand new smartphone that was supposed to arrive on your doorstep one week ago is something that most people would certainly appreciate. Moreover, we can use events to reconstruct past aggregate states to solve complicated issues or roll back unwanted changes.
If you’re like me before I knew anything about Event Sourcing, by now, you should have more questions than answers.
Why would one use Event Sourcing
Event Sourcing is by no means the silver bullet for software systems. Yet it can be an incredible tool if you know all the whens and the hows.
Event Sourcing provides a complete audit trail of business transactions. Once a transaction has happened, you cannot delete it or change it. You can use those stored transactions to determine the system’s state at any previous point in time by querying the events up to that point in time. It enables you to answer historical questions from the business about the system.
Troubleshooting is another fantastic Event Sourcing advantage. You can troubleshoot problems in a production system by taking a copy of the production event store and replaying it in a test environment. By troubleshooting, you might discover source code errors. Rather than performing risky manual data adjustments, you can fix the code and replay the event stream letting the system recalculate values correctly based on the new version of the code.
Event Sourcing often leads to better performance and scalability than complex relational storage models since you only use append-only operations for small immutable event objects. Having append-only operations allows you to reap the benefits of databases such as Cassandra. Moreover, Event Sourcing is usually combined with CQRS, which could be a crazy improvement on its own compared to the “classic” relational storage models.
Event Sourcing is an excellent fit in distributed systems (i.e., microservices architecture). You can publish events to notify other subsystems of changes to the application’s state in a considerably decoupled manner. Even though strictly speaking you can do that without Event Sourcing, having an event-based system makes it a lot easier.
A single one of those benefits could be handy on its own, but having them all combined often makes Event Sourcing a no-brainer.
Yes, yes, yes, but where is the code?
I wouldn’t be surprised if, at this point, you weren’t completely sold on the idea. The concept might seem difficult to grasp and hard to evaluate whether all of these benefits are real, or I am just a snake oil salesman trying to sell you my goods. Hopefully, the code will provide you with some answers.
The code examples are written in C# because its syntax is relatively simple and should not be challenging to understand for most developers. However, you could implement the same thing in many different programming languages, even the functional ones (see resources section).
Let’s define the Account aggregate model having some basic operations:
Instead of going for the “classic” way, we need to split each of our methods into two parts — one for validating the data and raising events, other for applying events. This separation is crucial since we’re going to restore the Account from our event store by applying events.
Let’s add some command handlers to orchestrate our application:
This is how we could implement the repository if we were to use Greg Young’s Event Store:
This is what the stored data looks like in the Event Store:
In a nutshell, the data is a complete log of the things that happened in the past. Now, whenever the business throws you a tricky “what happened” question, you can give a very detailed answer, feel like the smartest person in the room, and as a bonus, hopefully, get a raise.
Have you noticed the Version? The Version enables us to time-travel in the realms of Event Sourcing. To get a specific Account version, we can query related events from the database up to that version and use them to restore the Account. If using a version to query the data is not your cup of tea, you could use a Timestamp from your event store instead.
That’s it! That’s all there is! Well… not really… but by now, you should have a decent understanding and know quite well where to start!
Event Sourcing is not all fun and games
Event Sourcing pattern has a lot of good parts. However, there are some bad ones as well. If you think you need Event Sourcing — please read on.
Most of the developers are not familiar or not familiar enough with the Event Sourcing concept. Having to explain Event Sourcing to less experienced colleagues takes much time. Moreover, explaining only Event Sourcing is usually not enough. DDD, CQRS, eventual consistency model often goes hand-in-hand with Event Sourcing. It’s a very steep learning curve!
Most of the domains are not that complex. There, I said it. Event sourcing, on the other hand, is very complex. You have to consider snapshotting algorithms, not being able to change the data in the database in case there’s an urgent production issue, handling event schema changes, and that’s just the tip of the iceberg. Using Event Sourcing for a simple CRUD-based application is like using a chainsaw instead of a butter knife to make a sandwich — they both cut things, but one of them is a bit of an overkill. I will let you decide which one.
Even though the Event Sourcing pattern is great, you have to thoroughly consider whether it’s worth using it for your specific problem.
Frequently asked questions
As it turns out, Event Sourcing isn’t that straightforward as it might seem from the first sight. However, I find that some of the questions about Event Sourcing are asked more frequently than others.
Does Event Sourcing hurt the performance?
If done well, Event Sourcing doesn’t hurt the performance. It’s often quite the opposite since Event Sourcing usually comes together with the CQRS pattern, which is known to solve many performance issues. Of course, there are cases where aggregates tend to have long and complex lifetimes (having many events), making the querying part very slow, but that’s solvable with snapshots.
What’s the difference between domain and integrational events?
People often tend to split events into two categories — integration and domain, especially when working with microservices-based architecture. These categories are very similar since both types of events are state change notifications. However, the reasoning behind them is quite different. Integration events are for notifying other services about the state changes that happened inside of your service. Therefore, they must be changed as rarely as possible and might contain additional information that domain events don’t. Domain events are for notifying aggregates inside your domain boundaries (and restoring the state in Event Sourcing). Therefore, they can be changed easier as well as have only a minimal amount of relevant information.
What data stores can I use to implement an event store?
We can implement an event store using SQL Server, MySQL, Cassandra, Event Store, and many other data stores. We should be able to use our event store to query events in the same order as we produced them for a specified aggregate ID, save events atomically, utilize snapshotting. Other event store capabilities are mostly related to functional and non-functional application requirements. It’s worth mentioning that there are many debates on the internet discussing whether streaming platforms such as Kafka could be used to implement an event store. I’ve seen it work for specific business problems where there wasn’t much data stored, and heavy in-memory caching was used. However, in most cases, you’re better off with a different datastore since you would most likely need to query events for a specified aggregate ID in a reasonable amount of time.
Can I update stored events?
One day our event schemas are inevitably going to change. Therefore, it’s crucial to think about dealing with it ahead of time. When the inevitable happens, it becomes very tempting to change the stored data. However, most of the time, we shouldn’t. By doing that, we’re messing with historical data, which eliminates any guarantees of the data being authentic and makes it less accurate. What we should do instead is to version our event schemas. Event schema versioning allows us to adjust the code according to new business requirements while keeping the stored data unchanged. The mapping between old and new schemas could be hidden somewhere in the infrastructure layer of our application.
Can I use Event Sourcing without CQRS?
While it’s possible to use Event Sourcing without CQRS, it’s often not very practical. We usually need to query data from multiple aggregates combined. Moreover, it is relatively simple to build custom query models by subscribing to produced events. Event Sourcing and CQRS combination is like a pair of socks — you could go with one sock instead of two, but why on earth would you do that?
Resources
If you’re interested in learning more about Event Sourcing, here are some of the best resources you should start with.
Exploring CQRS and Event Sourcing: A journey into high scalability, availability, and maintainability with Windows Azure
This free e-book is hands down the single best resource that’s available right now about Event Sourcing. It covers a team’s journey to CQRS and Event Sourcing in a very detailed fashion with plenty of code examples and explanations of every decision. All of the code used in the book is available in GitHub. I can’t stress enough how good this book is!
Event Storming
Event Storming and Event Sourcing are entirely different things. However, they work exceptionally well together. Using the Event Storming method, we model our business domain on a whiteboard in aggregates, commands, and events. If done right, we can transform the whiteboard model to the source code with relative ease.
Simple CQRS
Simple CQRS is a sample project created by Greg Young to illustrate CQRS and Event Sourcing. While the code is not quite production-ready, the complexity is extremely low, making it a very nice example for beginners.
Domain-Driven Design, Event Sourcing and CQRS with F# and EventStore
If you’re not a big fan of OOP, you might enjoy trying the pattern in a functional language such as F#.
Final words
To use or not to use Event Sourcing, that is the question. Event Sourcing pattern indeed brings more additional complexity than the classic “store the current state” approach. On the other hand, it tends to bring many fantastic benefits such as performance gain, ability to restore the past states of your application, do state rollbacks, and, most importantly, tell exactly what happened if things were to go south. If used in the right place at the right time, Event Sourcing is going to be your best friend that you wish you had met sooner.
Thank you for reading. I’d love to hear your experiences with Event Sourcing.
