Data Modeling For Data Engineers — If you missed it, I had a live show a few weeks ago with one of the co-authors from Fundamentals Of Data Engineering defined the concept of query-driven data modeling. Now he was referencing the standard data modeling concepts which are: Conceptual Logical Physical but adding the 4th — Query-Driven-Modeling …

If you work in tech, then you know the curse of tutorial hell. You either hear about a new technology or finally convince yourself you’re going to learn about Kubernetes, Databricks, or Scala. …

Starting new data engineering projects can be challenging. It’s easy to get stuck on finding the right data and picking which tools you should use. And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult. Here were…

What’s the difference? — Data engineering and machine learning pipelines are both very different but oddly can feel very similar. Many ML engineers I have talked to in the past rely on tools like Airflow to deploy their batch models. So I wanted to discuss the difference between data engineering vs machine learning pipelines. …

By Daniel Beach author of Data Engineering Central Every once in a while, a new tool breaks into the Data Engineering space that bubbles up from underneath the festering masses, like a groundswell, pushing up through the thick layers of muck and mire on social networks. It’s like a fresh…

By Ian Schweer League of Legends and TFT(Teamfight Tactics) are worldwide games that serve millions of players. League of Legends is a 5v5 MOBA (multiplayer online battle arena) game where 10 players log into a game, each picking a champion for their particular role in the game (Top lane, mid…

Do You Know Them All? — Looking at the data engineering Youtube space 2 years ago. There weren’t that many people talking about the subject. There were a few but overall most data-focused videos were on data science, algorithms, and machine learning. But, somewhere in the middle of last year, it seemed like more and more…

Can AWS and Snowflake reduce data duplication? — AWS has jumped on the bandwagon of removing the need for ETLs. Snowflake announced this both with their hybrid tables and their partnership with Salesforce. Now, I do take a little issue with the naming “Zero ETLs”. Because at the very surface the functionality described is often closer to a…

60% of the respondents to my 2022 State Of Data survey aren’t confident they know what is required to get a promotion. This made me re-examine what I know about leveling and skill expectations. Seriously, what is expected at different levels? I have found that tech companies will break down…