A quick recap of the Event Sourcing concept

Event sourcing is a practice of storing data in a way that doesn’t include destructive updates, which opens up a lot of attractive possibilities.

Inspiration

Updating data “in-place”, aka “destructive updates” is “BAD” for two reasons: (1) we lose information, (2) no recovery from corruption.

So what do we do? We borrow from the experience of other industries. For example, accountants don’t ever store “current balance”. Instead they store a full history of debits and credits to the account, so that they can derive the balance at any point in time. Similarly with lawyers: they never update the contract “in place”. Instead, they add an “amendment”, which is basically a diff. Then they can derive the “current” state of the contract from those diffs. Or, we can even borrow from our very own industry: Git. Nuff said.

So the general idea would be this: we never persist “current state”, but instead persist “what happened”. Very importantly, we treat the “what happened” storage as absolute truth. It is a direct reflection of the real world. We cannot go back and “rewrite history”, because then it would be lying to us: it would not tell us what actually happened in the real world.

Photo by Belle Hunt

Aggregate

So, purely “mathematically”, we derive the “current” state of the world by folding over the whole history of events. This is nice. However, this may be prohibitively expensive. The world is big, the history is long. So we employ a specialization: we index events by certain integral parts of the world. These parts we call “aggregates” — they’re basically the same as “objects”, but we don’t want to call them that to avoid the stigma. :-)

For accountants, an aggregate is a single account. For lawyers — a contract. For Git users, the aggregate is the repo.

Now, when we need to derive current state of an aggregate, we can take events for only that aggregate and fold over them.

Snapshots

Sometimes, even rebuilding one aggregate instead of the whole world is not cheap enough. If my account existed for tens of years, and if I used it multiple times a day to buy groceries, the ledger is going to be huge. To mitigate this, we can store a special kind of event every now and then: a snapshot. This event will consist of three things: (1) aggregate id, (2) “current” aggregate state, (3) timestamp of the last event that existed at the time of the snapshot creation. The third component is important: it allows us to take our time computing the snapshot, and by the time we are ready to publish it, if there are already more events in the stream, we don’t care — those new events will just make it into the next snapshot. This means we can publish snapshot events without timestamp guard (see below), thus increasing performance.

Now, to rebuild the aggregate, we would take the latest snapshot plus all events after it, and fold over the events with the snapshot as the seed. Performance achieved! :-)

Timestamp guard (aka optimistic concurrency)

Sometimes we need to check the “current” state of the aggregate and issue (or not) some new events depending on the state. This can lead to race conditions: after I have fetched the current events, but before I’ve issued new ones, if somebody else has managed to issue more, then my decision is now based on outdated data, but I’ll never know this. To mitigate this issue, we employ optimistic concurrency: when issuing a new event, we also provide the timestamp of the latest event we know about, and if the event store knows about any newer events, it will reject our request, after which we can retry. Note that this is not necessary if my new events aren’t based on the current state.

Projections

In most systems, reads are way more frequent than writes. Having to rebuild an aggregate from the event stream on every read is going to affect performance. Plus, there can be cases where we need to traverse some non-trivial relationships between aggregates in order to show the data from some interesting perspective. Using the event stream for reads, therefore, is not ideal.

Instead, we create a separate database — for reads only. This database is continuously updated by a service that listens to the event stream and translates events into destructive updates of the database. Such database, together with the service that owns it, is called a “projection”.

Note that there can be multiple such databases (see “Advantages” below).

Lookup data

Not all data need to be represented as event streams. For example, if our application needs a list of all countries, it would be quite ridiculous to seed it with a bunch of “new country added” events, and then rebuild the list of countries from these events every time we need it. These kind of data can be managed via plain CRUD, provided one of the following:

1. Either these data are “append-only” (i.e. we never delete or update anything).
2. Or these data are not used in command validation (see below).

For example, the list of countries can be “append-only” if we only add countries, never delete them, and if a country happens to disappear from the world, we don’t delete it, but append a new “date of disappearance” datum to it. Or, the list of countries can be there for informational purposes only: just to display on the screen for the user, with no references from the events, and no decisions depending on existence or absence of a given country.

Commands vs. Events

This is an important distinction (so important it even got its own abbreviation).

A “command” is what somebody is asking me to do. An “event” is a record of what has already happened.

A “command” can fail: someone asked me to do something, I saw that I was unable to do it, so I refused. An “event” cannot fail: it’s a record of history, there is no arguing with it.

The relationship is not necessarily one to one: processing a command may result in issuing zero or more events. For example, if you ask an accountant to transfer some money from one account to another, the accountant will make two records — one debit and one credit. In practice, however, there is often a very significant overlap.

Command validation

When I receive a command, sometimes I need to make sure that it’s actually possible to carry out — aka “validation”.

We cannot validate commands against a projection — is a very important, yet often overlooked point. Because projections are eventually consistent (i.e. don’t get updated immediately to reflect new events), validating commands against their data may lead to race conditions.

Instead, command validation must happen against the event stream itself. Upon receiving a command, we rebuild the aggregate from the event stream (possibly with snapshots), validate the command against the aggregate, then issue appropriate event(s). If events fail to issue due to optimistic concurrency, we just repeat the whole process from the beginning: it will be as if we just received the command, the whole failed attempt never happened. Transactionality achieved! :-)

The nice upside of this is that, if a given command doesn’t require validation, we can just issue corresponding event(s) without a timestamp guard, and avoid extra blocking.

Advantages of Event Sourcing

1. Never lose data. Everything that happens is recorded forever, no data is erased by destructive updates.
2. Always have an audit trail to figure out how things got this way.
3. Since the event stream is append-only, it can be made very fast.
4. We can have multiple projections showing the system from different perspectives, possibly employing different kinds of databases, etc., while not stepping on each other’s toes.
5. If a projection’s data gets corrupted, there is always a last resort of rebuilding it completely by rerunning it from the beginning of history.
6. We get to come up with new kinds of projections “after the fact”, and still have them encompass the whole history, all the data.

Disadvantages of Event Sourcing

1. Eventual consistency: projections don’t get updated immediately when events are issued. Doesn’t really matter for most applications.
2. Increased system complexity. This is a very debatable point. For very toy, all-CRUD applications, event sourcing is indeed a bad fit, but most real-world applications are not like that at all, and the complexity of managing destructive updates outweighs the complexity of event sourcing very quickly.
3. Increased space consumption (i.e. storing the whole history instead of just the latest state). This has proven to be a non-issue in practice. Just look at Git.