Events are the Key to Master Data
Events are the most powerful type of data. Unfortunately, it is also the most underrated. Why are events so essential, and how can we make better use of them? Let’s have a quick look.
Data has a History
When people think about data, they usually focus on mutable data, on entities, on domain objects. They see data as a collection of information with a state that evolves. And in many applications, it is most likely the only type of data you will find. But when you change the value of your data, you lose data. You lose the value it had yesterday. You lose the transformation that you applied to it. You lose the history of your data. And when an error occurs, there is no going back.
Fortunately, the solution is straightforward: use events! Events are also data, but they are immutable data. An event represents a transformation to the state of your application. It is similar to a transaction in a ledger. If you want the state at a given time, you just need to evaluate all the events until that specific time. Saving all the events that happened in your system creates a full history of your data. And when an error occurs, just go back in time.
Data has a Context
Another essential aspect that is very often neglected is the context of your data: Who modified your data? Why did it get updated? When did it happen? When you save a history of your data, you are creating a context. You are not only keeping a backup of your data to prevent it from getting lost or corrupted. You are building a context that will increase knowledge, comprehension, and predictions. The context gives us information about behavior, patterns, habits, and much more.
Errors are also events, and they should be part of your data too. Failures are a crucial part of the context. They will help you understand what went wrong, how to fix it, and how to make improvements. Triggering events when an error occurs can help add additional context and enable your system to react and respond to failure.
Data in the Real World
If we take a step back and observe the world around us, we will quickly realize that our lives are sequences of events. We live in a dynamic world that is continually evolving around us. When we solve a problem, our memory is not only remembering the solution but also how we solved it. Our brain doesn’t only remember the latest state of our surroundings; it keeps the sequence of events.
Captured data in the form of events is a natural and logical way to represent our surroundings. It will not only improve the quality of our data, but it will also bring the data closer to the real world and make it more accurate.
How to use Events
You have different options to make use of events. Events are not limited to microservices; you can use them in monoliths too. You don’t have to use events everywhere, but the more you use them, the more data you will collect.
Event sourcing is the most famous architecture when it comes to events. But even if you make a simple CRUD application, you can record events. If you create a new User, record a UserCreated event. If you update a BankAccount, record a BankAccountUpdated event. And don’t forget to capture essential information: the date and time, the id of the user who triggered the action, was the operation a success or an error.
We will cover more about this subject in upcoming articles 🤓. As a data engineer, I will mainly focus on how events can have a significant impact on data science. Meanwhile, I highly recommend you to have a look at Domain-Driven Design¹ from Eric Evans, to explore more about the concept of Domain Events:
“A domain event is a full-fledged part of the domain model, a representation of something that happened in the domain.”
And of course Event Sourcing² from Martin Fowler:
“Event Sourcing ensures that all changes to application state are stored as a sequence of events.”
Conclusion
Don’t underestimate the power of events. Working with events is like working with a notebook: you can work without it, but why would you want to make your life harder? Being able to have a clear history and context will save you a lot of time and mistakes.
References
[1] Eric Evans (2015). Domain-Driven Design Reference https://domainlanguage.com/wp-content/uploads/2016/05/DDD_Reference_2015-03.pdf
[2] Martin Fowler (2005). Event Sourcing https://martinfowler.com/eaaDev/EventSourcing.html
