How Data Engineers Win (Discipline and Fundamentals)
The Real Things Every Data Engineer Should Care About
Data is a land of opportunity. It’s also the land of choice.
Too much choice, if you ask me. Too much choice leads to confusion. Too much choice breeds too many opinions. That is when common sense goes right out the window.
Example:
Airflow, Dagster, Prefect, Mage AI, Kubeflow, Flyte, and Luigi. This list reads like some kid’s favorite Pokémon.
Here’s the secret about most tools in Data:
They all do the same thing! Some do it better. Some have extra bells. Some have fancy whistles. Some are cheap (this is what people care about), most are expensive — like crazy expensive. Some are easier to use than others, and all have the world’s leading marketing teams trying to ram it down your throat the first chance they get.
End of the day, they all do the thing it says on the tin. Take the above list, for example. They all do the same thing — orchestrate. End of story.
Here’s another example: BigQuery vs. Databricks. Databricks vs. Snowflake. Snowflake vs. Redshift.
On and on it goes.
