Amdahl’s Law for Data Pipelines

Fabian Nick
Dec 15, 2021 · 5 min read

In the past couple of months and years, I’ve come to notice how often data projects become incredibly hard or fail because the data pipeline isn’t working correctly. Most of the time this is because, at some point in the pipeline, human intervention is necessary.

TL;DR: Keep any human intervention out of your data pipeline as soon as possible. It will make the operationalization of your solution incredibly hard.

In this article, I will draw some parallels between this observation and Amdahl’s Law, which is often cited when talking about the maximum…