What You Should Consider When Working With Airflow — Airflow remains a popular choice when it comes to open-source orchestration tools. When I surveyed people about a year ago now, it was the most popular open-source solution, and still to this day, my video on “Should You Use Airflow” drives a lot of prospect conversations. Now, I do want…

Data Modeling For Data Engineers — If you missed it, I had a live show a few weeks ago with one of the co-authors from Fundamentals Of Data Engineering defined the concept of query-driven data modeling. Now he was referencing the standard data modeling concepts which are: Conceptual Logical Physical but adding the 4th — Query-Driven-Modeling …

If you work in tech, then you know the curse of tutorial hell. You either hear about a new technology or finally convince yourself you’re going to learn about Kubernetes, Databricks, or Scala. …

Starting new data engineering projects can be challenging. It’s easy to get stuck on finding the right data and picking which tools you should use. And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult. Here were…

What’s the difference? — Data engineering and machine learning pipelines are both very different but oddly can feel very similar. Many ML engineers I have talked to in the past rely on tools like Airflow to deploy their batch models. So I wanted to discuss the difference between data engineering vs machine learning pipelines. …

By Daniel Beach author of Data Engineering Central Every once in a while, a new tool breaks into the Data Engineering space that bubbles up from underneath the festering masses, like a groundswell, pushing up through the thick layers of muck and mire on social networks. …

By Ian Schweer League of Legends and TFT(Teamfight Tactics) are worldwide games that serve millions of players. League of Legends is a 5v5 MOBA (multiplayer online battle arena) game where 10 players log into a game, each picking a champion for their particular role in the game (Top lane, mid…

Do You Know Them All? — Looking at the data engineering Youtube space 2 years ago. There weren’t that many people talking about the subject. There were a few but overall most data-focused videos were on data science, algorithms, and machine learning. But, somewhere in the middle of last year, it seemed like more and more…

Can AWS and Snowflake reduce data duplication? — AWS has jumped on the bandwagon of removing the need for ETLs. Snowflake announced this both with their hybrid tables and their partnership with Salesforce. Now, I do take a little issue with the naming “Zero ETLs”. Because at the very surface the functionality described is often closer to a…