This post was co-written with Mike DiScala.
Event sourcing is at the core of our data pipeline: each night we extract data from our client systems and construct an event log describing everything that happened within the data (e.g., a student was added, attendance was taken, grades were recorded). We replay these events in our code to construct Postgres tables that our application queries to answer questions like “What is student XYZ’s name?” and “How many days of school has student XYZ missed this year?”
In event-sourcing parlance, these tables are called projections. Even though we only add one day’s worth of events each night, we re-project the entire history of events daily to ensure we can always change a projection’s interpretation of a past event — or even build new projections over our complete event stream. …
At Panorama, we strongly believe in automated tests. No code gets deployed without passing both a thorough code review and a battery of thousands of automated tests, and no code makes it through code review without updating tests for the new features it’s adding. Automated tests help us build new features and refactor old code without introducing bugs, and are a big part of the reason we are able to confidently deploy new changes to our production apps multiple times a day.
But with an automated test suite comes the dreaded possibility of tests that fail nondeterministically-that is, most of the time they pass, but every once in a while they fail for no obvious reason. …
As Rails developers, we spend a fair amount of time in the Rails console, which is itself just the native Interactive Ruby Shell (IRB). By default, it’s not particularly pretty…
$ rails c
irb(main):001:0> def hello
irb(main):002:1> puts “hello”
…but it gets the job done. We were content with it, because we didn’t know things could be better.
But then we started worrying. Our app lives on Heroku, and there are several other Heroku “forked” apps that give us non-production environments for various levels of testing. And with many shells open at once, it could be possible to accidentally run code against our production database that was intended for casual testing with fake data. (We have code reviews for scripts that modify production data, but what if the script was only ever supposed to be run locally?) …