Event Sourcing - Step by step in F#

This article is part of traditional #fsadvent. Thanks all the writers involved in it and of course thanks Sergey for organization!

Why Event Sourcing again?

Well, looking back at this year agenda, Event Sourcing was somehow the thin red line of me and F# in 2018. Yes, I could write about few more things I did this year like writing my first type provider for OpenAPI specification (type provider has always been in my mind as something like graduation for F# developers, so hurraayyy), starting new F# commercial projects in our company, co-organizing LambdUp conference, F# workshops, public talks in Czech Republic, Slovakia, United Kingdom and Germany or organizing another year of my beloved FSharping meetups in Prague.

But still, most of the things were connected with this year’s major topic: Event Sourcing in F#. I learnt a lot and now feel this article could be great way how to summarize it and also leave it for a while. I mean… not leave as stop with Event Sourcing (God, no way! :)), but more likely to move on to more advanced topics like Process managers, Sagas, queues, compensating events, etc…

[Roman; Roman] |> Async.Parallel

Before we start, there is one more thing I need to mention. On FableConf 2018 / RemmiDemmi (frankly, I still don’t know how I got there, but thanks Alfonso, Steffen and others for lack of sound judgement to invite me) I met very pleasant and friendly Roman Sachse a.k.a. R0mmsen who’s talk was focused on the same topic. We found another funny similarities (except first name, obviously) like having two daughters, but back to my point: There is another F# developer doing the same thing, which is absolutely great! We can share/compare our approaches to the same problems and learn from each other. His FableConf talk was amazing (waaaayyy better than mine) and I’m truly looking forward for Roman’s #fsadvent contribution. It should be out on 23rd of December so don’t miss it!

Let’s start then, can we?

Ok, no more intros, let’s start. So what really is Event Sourcing? Before we got to the famous one-sentence explanation, we need to define what Event is.

Event

If you expect something difficult and complex, I have a bad news for you. The definition of Event is straightforward:

Event is fact that happened in past.

Yeap. That’s it. No rocket science, is it? Just thing that happened. Like you sent an email, published article or locked door. It is just done and cannot be changed. I mean, you can change it by doing another action (unlocked door), but cannot change the fact, that you locked them first. We have name for this. Yes, it is the famous, the one and only… drum rolls… immutability!

Protip: If you don’t believe me, try to fire my favorite Event called SomethingInappropriateSaidToWife and change it day after. Good luck & have fun.

Ok, now we know what Events are, and that they are immutable, but how they come into existence?

Command

Whenever you say “Command”, somebody will raise hand and say: “But, but, now we are getting into CQRS!” Yes, you are right my meticulous friend. We are. I believe the efficient Event Sourcing is tightly connected with CQRS. You can probably avoid it, if you must, but I don’t see any point in doing this so I’ll happily mix terms Event Sourcing & CQRS together.

So, what is Command? Again, no rocket science here:

Command is an instruction to your system than can lead to new Events.

Keeping the simple example of door, it can be: LockDoor, UnlockDoor, OpenDoor or CloseDoor. The reason why Commands exist is that you want something to happen, BUT!, there is no guarantee it will succeed.

There is no door, Neo.

You can try to fire Command CloseDoor here, but instead of DoorClosed event you would probably get something like ThereIsNoDoorException.

Commands mean intention. Intention without guarantee of success. If you are lucky and your system is in correct shape, you’ll get new Events. If not, you’ll get an error.

Events without Commands

Commands are not the only way, how Events come into existence. Sometimes you don’t want to validate or even create Commands, because it doesn’t make sense to do so. From technical or business perspective. For example storing Events from IoT temperature sensor will go without Commands. Mother nature doesn’t ask us: “Hey, I see you are wearing only t-shirt so I’d like to set outside temperature to -25°C, can I?” You cannot reject that temperature has already changed.

The same goes with business reasons. If you implement some kind of e-shop, you don’t want to stop business only because you don’t have something on stock. You can sell it now and figure out the delivery later. You can always return money to the customer. It sounds strange, but go on some well known e-shop website and try to actively not buy something. Business must go on!

Back to Event Sourcing…

So, we have Commands and Events, now it is finally time for the famous one-liner you have been desperately waiting for:

Event Sourcing is storing the system changes itself instead of result of such changes.

Again, easy as taking lollipop from small kid (please don’t do it, you ruin their childhood, ok?!). Instead of writing UPDATE SQL on database layer, you just store the change (Event) and do nothing. I promise, you’ll get to your ORMs /Entity Frameworks /other cute DB things you love so much later but for now just store all the event data. But where to store them?

Event Store

Event store is the answer. Special, append only database designed for storing (appending) Events. Data stored there are not organized in tables like in SQL databases, or collections like in document databases, but in something called Streams. Imagine Stream as drawer where you put all related Events one-by-one as they occurred. You probably already realized, but just to be sure — you cannot delete Events from Event Store. Once you store your Event, you have to live with it. You can migrate them to different Stream later, but migration/versioning of Event Sourcing systems is so complex topic, even Greg Young hasn’t finished his book yet, so we will skip it.

<self-promo>
If you need Event Store on Azure Cosmos DB or Azure Table Storage, give a shot mine implementation called CosmoStore. You can find it on https://github.com/Dzoukr/CosmoStore
</self-promo>

Once you have Event Store we can start coding, because we have everything we nee… oh wait!

State

Observant readers already realized in Commands sections, that I missed something important there. Yes, in the Command validation part. I intentionally missed how and against what we validate our Commands, because first we needed go through all previous chapters.

The last missing part of all of this Event-sourcing-monster-parade is called State or Projection.

State/Projection is structural view of impact of Events.

Sounds fancy, but what is it really? Going back to simple-stupid example of opening/closing door, the State here is the door. In other words in what “condition” is door after all Events? Locked, opened, closed, broken? That is the State. And that is the thing we use as internal representation of result of Events used in Commands validation.

Ok, enough talk and theory, let’s go write some code!

But before that, take 47 seconds break…

…and that’s why I don’t like cricket!

F# Example

We are going to write simple TodoMVC application. We will add tasks, remove tasks, mark them as completed and so on. We gonna use all the information gained from previous chapters (except Bob Fossil’s dance) to build nice little Event Sourcing based app.

Commands

We gonna start with commands. In F# we have a union which is perfect data type for such modelling:

type Command =     
    | AddTask of CmdArgs.AddTask    
    | RemoveTask of CmdArgs.RemoveTask    
    | ClearAllTasks    
    | CompleteTask of CmdArgs.CompleteTask    
    | ChangeTaskDueDate of CmdArgs.ChangeTaskDueDate

Each case holds event-specific data so we need to define it in separate module:

module CmdArgs =    
    type AddTask = {        
         Id : int        
         Name : string        
         DueDate : DateTime option    
    }     
    type RemoveTask = {        
         Id : int
    }     
    type CompleteTask = {        
         Id : int    
    }     
    type ChangeTaskDueDate = {        
         Id : int        
         DueDate : DateTime option    
    }

Tip: RemoveTask and CompleteTask has the same structure, but keep them as separated types. The fact they look the same can change in future and your future refactoring-all-the-stuff-yourself will thank you later.

Events

As you can see, Events look almost the same as Commands, but are in past tense.

type Event =    
    | TaskAdded of CmdArgs.AddTask    
    | TaskRemoved of CmdArgs.RemoveTask    
    | AllTasksCleared    
    | TaskCompleted of CmdArgs.CompleteTask    
    | TaskDueDateChanged of CmdArgs.ChangeTaskDueDate

The data connected to each case are exactly the same we have for Commands, but don’t take this as rule of thumb. Sometimes it makes sense to have slightly different values for Events, but it is easier to start like that and change it later, if really needed.

State

What is the State for TodoMVC apps? Yes, it is simply the list of tasks:

type Task = {    
    Id : int    
    Name : string    
    DueDate : DateTime option    
    IsComplete : bool
} 
type State = {    
    Tasks : Task list
}
    with static member Init = { Tasks = [] }

We have basic domain of our application (State, Commands, Events), so now we add few more things and glue everything together.

Tip for lazybones: Copy-paste from my talk example app.

State Init (a.k.a Empty State)

How does State look like when you start with no tasks? Yes, empty list of tasks. We will need it later when applying all the Events on state (you must start somewhere). Because I am incredibly lazy, I already did this on type as static member Init so all the work here is done.

Execute function

Execute function is the place where Events are created based on Commands. You already know that there is no guarantee it will succeed, so it’s the perfect place for validation. And yes, you guess right, we are validating against the latest State so this is why we need it as (the first) parameter.

The function signature is 'state -> 'command -> 'event list knowing that function itself can throw exception. If you are more into ROP (which I am a lot), you can change signature a little bit:

'state -> 'command -> Result<'event list, YourDomainError>

Here is the beauty:

let execute state command =     
    let event = 
        match command with    
        | AddTask args -> 
            args.Id 
            |> onlyIfTaskDoesNotAlreadyExist state 
            |> (fun _ -> TaskAdded args)    
        | RemoveTask args -> 
            args.Id 
            |> onlyIfTaskExists state 
            |> (fun _ -> TaskRemoved args)    
        | ClearAllTasks -> AllTasksCleared    
        | CompleteTask args -> 
            args.Id 
            |> onlyIfTaskExists state 
            |> (fun _ -> TaskCompleted args)    
        | ChangeTaskDueDate args -> 
            args.Id 
            |> (onlyIfTaskExists state >> onlyIfNotAlreadyFinished)
            |> (fun _ -> TaskDueDateChanged args)
    event |> List.singleton //we must return list of events

Add few more validation functions. To keep examples simple, I’ll stick with throwing exceptions, but in real-worlds apps we use custom domain error discriminated union:

let onlyIfTaskDoesNotAlreadyExist (state:State) i =    
    match state.Tasks |> List.tryFind (fun x -> x.Id = i) with    
    | Some _ -> failwith "Task already exists"    
    | None -> state 
let onlyIfTaskExists (state:State) i =    
    match state.Tasks |> List.tryFind (fun x -> x.Id = i) with
    | Some task -> task    
    | None -> failwith "Task does not exist" 
let onlyIfNotAlreadyFinished (task:Task) =    
    match task.IsComplete with    
    | true -> failwith "Task already finished"    
    | false -> task

And we have new Events created based on each Command including domain validation! *silent high five*

Apply function

Apply function is the place where you affect you State (the list of task in this case) by applying each change on State. In other words: How does Event XYZ change your State.

The signature of Apply function is 'state -> 'event -> 'state and must never throw an exception.

let apply state event =     
    match event with    
    | TaskAdded args ->         
        let newTask = { 
            Id = args.Id 
            Name = args.Name 
            DueDate = args.DueDate 
            IsComplete = false
        }        
        { state with Tasks = newTask :: state.Tasks}    
    | TaskRemoved args -> 
        { state with Tasks = state.Tasks |> List.filter (fun x -> x.Id <> args.Id) }    
    | AllTasksCleared -> { state with Tasks = [] }
    | TaskCompleted args ->         
        let task = 
            state.Tasks 
            |> List.find (fun x -> x.Id = args.Id) 
            |> (fun t -> { t with IsComplete = true })        
        let otherTasks = state.Tasks |> List.filter (fun x -> x.Id <> args.Id)        
        { state with Tasks = task :: otherTasks }    
    | TaskDueDateChanged args ->         
        let task = 
            state.Tasks 
            |> List.find (fun x -> x.Id = args.Id) 
            |> (fun t -> { t with DueDate = args.DueDate })
        let otherTasks = 
            state.Tasks 
            |> List.filter (fun x -> x.Id <> args.Id)        
        { state with Tasks = task :: otherTasks }

Looking at all three functions (actually two plus one initial value), we can wrap it in universal generic type:

type Aggregate<'state, 'command, 'event> = {    
    Init : 'state    
    Apply: 'state -> 'event -> 'state    
    Execute: 'state -> 'command -> 'event list
}

CopyPastaTip: Be lazy, copy directly from another example.

Before we continue…

I think this is the great moment for little summary. So where we are?

We have:

• Domain (list of tasks)
• Commands (way how to add/remove/complete tasks)
• Events (results of our Commands)
• Function for creating Events based on Commands (Execute) with basic built-in validation
• Function for creating State based on previous Events (Apply)
• Initial (empty) State we will start with
• All wrapped in Aggregate type.

let tasksAggregate = {
    Init = State.Init
    Execute = execute
    Apply = apply
}

Great time to finally involve Event Store.

Event Store Command Handler

From the F# signature perspective the Event Store Command Handler looks almost like Execute function, but it has few dependencies “baked-in” or partially applied as parameters:

• Event Store implementation
• Aggregate type with Init, Execute and Apply functions

The Command Handler flow is:

1. Get the Command as parameter
2. Load all previous Events from Event Store Stream ordered by time they occurred
3. Use Apply function to affect Init State, one by one. Just a typical fold function.
4. Use the latest (current) State (result of folding/applying Events from previous step) as parameter for Execute function + use Command we got as parameter
5. In case we got error, we end here and return/throw error
6. In case Execute went ok, we now have new Events
7. Save (append) those new Events back to Event Store
8. Return new successfully created and stored Events

Looking at another example file, the quick-and-dirty sync version can be written like this:

// create new Event Store
let store = CosmoStore.CosmosDb.EventStore.getEventStore myConfig
// get current state from Tasks stream
// note the use of tasksAggregate record
let getCurrentState () =        
    store.GetEvents "Tasks" EventsReadRange.AllEvents        
    |> Async.AwaitTask        
    |> Async.RunSynchronously        
    |> List.map (fun x -> Mapping.toDomainEvent (x.Name, x.Data))
    |> List.fold tasksAggregate.Apply tasksAggregate.Init
// append new events to Tasks stream
let append events =        
    events         
    |> List.map Mapping.toStoredEvent        
    |> List.map (fun (name,data) -> { Id = Guid.NewGuid(); CorrelationId = Guid.NewGuid(); Name = name; Data = data; Metadata = None })        
    |> store.AppendEvents "Tasks" ExpectedPosition.Any        
    |> Async.AwaitTask        
    |> Async.RunSynchronously        
    |> ignore
let handleCommand command =     
    // get the latest state from store    
    let currentState = getCurrentState()    
    // execute command to get new events    
    let newEvents = command |> tasksAggregate.Execute currentState    
    // store events to event store    
    newEvents |> append    
    // return events    
    newEvents

Note: Please check the example file for boring parts like transforming JSON payload to Events and back.

Congrats! You have fully working Event Store Command Handler! You can send Commands and get back new Events, which are stored in Event Store so your internal State is actually really changing! *loud high five*

But what about if we want to check current state? Do we have to go to Event Store, replay all Events, and create latest state to see the list of tasks? Well, we can but we will do something different.

Read (query) side

Do you remember me promising you’ll get to your fancy database stuff like Entity Framework, Dapper, SQLProvider, whatever…? So here it comes.

The read (also known as query) side is the thing you’ll use for reading data in “traditional way” — from relational database. Technically it is another projection you build as reaction on Events in you system and you build it with your favorite database library/framework. It’s like Apply function but on different— database — layer. This is the place for making CREATE, UPDATE, DELETE SQLs and other familiar stuff.

Just have one and the only concern in mind: “It must be easily readable/queryable”. Nothing else. That is the only reason for existence of Read side. And don’t get too attached to it. You gonna destroy/rebuild it many times, because once you are in Event Sourcing, you can always replay old Events, drop old database and create brand new projection stored somewhere else, in different structure, database, file… you name it.

In our system we usually built projections with function called *Event Handler and we have many of those. DatabaseEventHandler, EmailEventHandler, FileEventHandler, … Signature of such handler is (again…?) pretty easy: 'event -> unit

Need to print Events to console?

let handleEventToConsole event = 
    match event with
    | TaskAdded args -> printfn "Console handler says: Hurrayyyy, we have a task %s!" args.Name    
    | TaskCompleted args -> printfn "Console handler says: Task with ID %A is completed" args.Id    
    | AllTasksCleared -> printfn "Console handler says: ...and now they are all gone"    
    | _ -> ()

Need to work with SQL database?

let handleEventToSql = 
    match event with
    | TaskAdded args -> args |> sqlInsert    
    | TaskCompleted args -> args |> sqlUpdate    
    | AllTasksCleared -> sqlDeleteAll()    
    | _ -> ()

And glue them together to one handle function:

let handle evn =    
    evn |> handleEventToConsole    
    evn |> handleEventToSql

AnotherAmazingTip: CTRL+C, CTRL+V from just another example from still the same talk I am referencing to.

Final pipeline

It’s the final pipelineeeeeeee, 🎵🎵, tududuuuu, dudu, *air guitar*
…ok where was I?

Final pipeline is the function you’ll use in your Console/Giraffe/Suave/Saturn application as composition of Event Store Command Handler and Event Handler(s).

// simple function composing command handler + event handlers
let pipeline cmd =    
    cmd 
    |> CommandHandler.handle
    |> List.iter ReadSide.handle

So that’s all kids. Really, there is nothing else. Go home now. Thanks for attention and see you next time. Questions?

…Ufffff, ok. You, girl in brown sweater.

Q: Why should you even think about writing apps this way?

A: There are many advantages of such approach:

• You never lose any data. You just store all the data you have about Events and decide later (can be 1ms after, but 1 year as well) what kind of information you are interested in.
• Absolutely DDD friendly. Design your domain the way you think it should be and define Events around.
• Thinking and talking in Events makes communication with customer much easier. They don’t understand models, relations, constraints or foreign keys, but they will tell you after OrderReceived there should be EmailSent. You are now talking in same language.
• You get fully trustworthy audit log by design.
• You can replay old Event and get new information from them. For marketing, security or whatever reason (somebody said “bugfixing”?), you can go through all the Events from the very beginning and create new projections.

…Great, another one. You, little boy with Java is GR8 t-shirt.

Q: Can we rename Command Handler to IBusinessDomainEventFactoryProxyImplProvider?

A: Awesome question, thanks for that! Absolutely not.

…next?

Q: Why you shouldn’t do Event Sourcing?

A: There are many reasons why to avoid this approach:

• Like it or not, the complexity of application is much higher than simple CRUD against SQL table.
• Restoring State from stored Events can be expensive. Speaking about thousands of Events in single Stream, it can take some time. If your application is performance-first + invalidating State every few milliseconds (gaming apps?), Events Sourcing is not for you.
• Until you get all your read side data from Event Store (which you still can, but can be ineffective), you are in the world of eventual consistency.
• Events migrations / versioning is not piece of cake.
• Correct splitting domain into aggregation roots and storing in Streams is probably the most difficult part I can think of.

…last one? Really, guys, I need to go trolling NuGet team about reinventing lock file, so time for the last question.

Q: You’ve sold me Event Sourcing, where should I go next?

A: If you need book, the great one is Domain Modeling Made Functional by Scott Wlaschin. Once you get deeper you really appreciate (not completed yet, sic) Versioning in Event Sourced systems by Greg Young who also gives great talks about CQRS & Event Sourcing. In F# community there is amazing Roman Sachse’s talk from FableConf 2018 RemmiDemmi track. And if you are in (or close to) Czech Republic we will be happy to welcome you at our F# meetups called FSharping where we can discuss further.

Thank you for reading and Marry Christmas to all of you! See ya in 2019!

*running frightened to car and drifting away*