ES1. Event Sourcing — Background Information

Over the last few years, I have been the chief architect of the Jumla platform which connected thousands of suppliers and consumers in the food industry. One of the key aspects of the design of Jumla was the adoption of event sourcing as the cornerstone of the architecture.

What is Event Sourcing?

Event sourcing is the organization of all of a system’s data as an append-only sequence of immutable events. An event denotes information about something that happened in the past. At first, there is nothing, then the first event happens, then the second, and the third, and so on, and so forth, until the end of time. Since events are about things that happened in the past, we cannot delete them, change them, or change their order.

Let’s take a simple example of an events type definition in Rust:

enum Event { 
  UserAdded {user_id: String, user_name: Option<String>},
  UserNameChanged {user_id: String, user_name: String},
  UserProfileUrlChanged {user_id: String, profile_url: String},
  UserRemoved {user_id: String},
}

Just by looking at the type definition, we can probably infer that this is a system in which

1. there are users who get added, possibly with a specified name, and
2. users are identified by their user id, and
3. each user has a name (optional) and a profile url (also optional), both of which can change over time, and
4. users eventually may get removed.

Here is a possible sequence of events that can happen in a system, each expressed in a JSON record:

{"name": "UserAdded", data: {"user_id": "user1", "user_name": "Alice"}}
{"name": "UserAdded", data: {"user_id": "user2"}}
{"name": "UserNameChanged", data: {"user_id": "user2", "user_name": "Bob"}}
{"name": "UserProfileUrlChanged", data: {"user_id": "user2", "profile_url": "https://bob.example.com"}}
{"name": "UserNameChanged", data: {"user_id": "user2", "user_name": "Bobby"}}
{"name": "UserRemoved", data: {"user_id": "user1"}}

Note that we are only explaining the concept, not the implementation, of event sourcing. What is important is that the events are recorded in a durable immutable storage and that they can be retrieved sequentially in the order in which they were added.

Event sourcing does not mean that we have to let go of a relational database model. On the contrary, event sourcing can be used to populate the data in a relational model. By reading the events sequentially, and performing certain rules associated with each event type, one can build the database, from scratch, using the events that were recorded.

In an event sourcing system, the events exist separately from the database and the data in the database is derived purely from the events.

Why Event Sourcing?

When designing a brand new system, it is often the case that the full requirements of the system are not known in advance. In the case of Jumla, we were to build the platform with minimal features (an MVP) that are known at the beginning, but the system had to be flexible enough to accommodate all yet unknown future features.

Traditionally, and as I have taught in university courses in the past, one would start with taking the functional requirements of the system and translating them into a relational database schema. Cool, but what happens tomorrow when the requirements change? What if the changes happen on daily basis? How do you rapidly implement these changes with no downtime and without losing any data?

System evolution is a hard problem using a traditional relational databases. When the database schema changes, we often write one-shot migration scripts. A migration script may be written in SQL or in any other language. The purpose of the migration script is to create, alter, or drop tables from the database, and move data around. Once the migration script is run to completion, one would just hope that all went well and that no data is lost or corrupted. This is very problematic even if the migration script was heavily tested on non-production data. To restate Murphy’s law: if you keep on doing migration scripts, eventually one will go wrong.

With event sourcing, the problem of system evolution is solved in a totally different way. Instead of modifying the production database A, what we do is build a new database B from scratch. Database B exists separately from, and in parallel to, database A.

Two databases are built in parallel using the same set of events.

What event sourcing gives us is the ability to do A/B testing on the primary production database. If the new processing rules (B) are deployed and they produce the desired database (B), we can swap A and B and voila, we have migrated the database to the new system. If, on the other hand, the new processing rules were found to be erroneous, and they did not produce the desired outcome, the database B can be dropped, completely, then the rules are fixed and the cycle repeats. There is never any danger of corrupting the database because the events themselves are immutable.

Other advantages?

The section above highlights the main advantage of using event-sourcing; one that sold me on events as a core architectural foundation for Jumla. There are other advantages of course and they may be equally important. These advantages include:

1. Events are processed in real-time leading to an overall real-time system.
2. All database records are annotated with information about the events that made them what they are, leading to getting record audits for free.
3. One can process events up to a certain point in time, and observing the state of the system at that time. The database is a “time-machine”
4. Deriving new data and information that are not conceived in the original design.

This article is the first of a series of articles that I’m planning to write on event sourcing. I plan on diving deeper into (1) the concepts, (2) the systems-level implementations, and (3) the application-level details and programmer experience. If you have any questions or feel that a certain topic needs further elaboration, please feel free to ask.

Part 2: Building a Simple Event Store