Why Your Data Pipelines Will Fail On These 10 Days Every Year (And What To Do About It)

When the clock strikes midnight on certain days of the year (or decade) your pipelines will transform into functionless code.

Currently job searching? Give yourself an edge by developing a personal project using my free 5-page project ideation guide.

One of my favorite shows in the olden times when the History channel still showed history was Life After People. Unlike any other History channel show at the time, like the title suggests, the series doesn’t feature a single human being. Instead, the show imagines what would happen to civilization if, one day people just… vanished.

Since each episode was an hour long, the show was able to extend pretty far into the future, with each segment beginning the same: “(Insert years) after people.” I remember watching everything from a time-lapsed, abandoned Central Park become overgrown with weeds to watching the inevitable befall fish trapped in aquariums with failing pumps.

The main argument of the show was that for all the complexity and scale of human civilization, the majority of infrastructure would fail within the first 30 days of neglect.

Time can have a similar impact on automated code, a.k.a. data engineering pipelines. No matter how many upstream checks you’ve built in or unit tests you’ve suffered through, there are times when…