What To Consider When Building Data Pipelines
The Baseline Data Stack Part 3
In 2021 we watched Fivetran raise $565 million, Airbyte $150 Million, Matillion $100 million, Rivery raised $16 million and Informatica went public.
All of these companies have some piece of their business connected to data pipelines. Also sometimes referenced as ETL, ELT, E(t)LT, and CDC.
For today when I say data pipeline I am focused on batch processing and what you need to consider when building batch data pipelines.
Regardless of the tools, you are using.
When Building Pipelines What Should You Consider
Tools and technology are just that.
🛠️ Tools.
They won’t actually drive any form of impact on their own.
They won’t develop processes that are connected to dashboards that in turn drive actions without people. Nor are the numbers they are creating going to magically jump off the screen and fix a business.
So before building any data pipeline it’s important to consider a few things.
