Image for post
Image for post

This post was co-written with Mike DiScala.

A scaling challenge

Event sourcing is at the core of our data pipeline: each night we extract data from our client systems and construct an event log describing everything that happened within the data (e.g., a student was added, attendance was taken, grades were recorded). We replay these events in our code to construct Postgres tables that our application queries to answer questions like “What is student XYZ’s name?” and “How many days of school has student XYZ missed this year?”

In event-sourcing parlance, these tables are called projections. Even though we only add one day’s worth of events each night, we re-project the entire history of events daily to ensure we can always change a projection’s interpretation of a past event — or even build new projections over our complete event stream. …

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store