Homepage
Open in app
Sign in
Get started
Making Sense of Data & Helping Others Grow: Tips, Advice, and Stories from the Front Lines of Data Engineering
MOST READ
WRITE FOR US
FEATURED ARTICLE
LinkedIn
Follow
SQL: You Can’t Be a Data Engineer Without It (Non-Negotiable)
SQL: You Can’t Be a Data Engineer Without It (Non-Negotiable)
Why SQL is Essential for Your Career
Tim Webster
Aug 29
Can You Crack This SQL Query in 30 Minutes?
Can You Crack This SQL Query in 30 Minutes?
Put your SQL skills to the test with this engaging, time-bound challenge.
Santosh Joshi
Aug 29
Best Orchestration Tools Besides Apache Airflow
Best Orchestration Tools Besides Apache Airflow
Orchestration Tools You May Not Have Heard of
Nnamdi Samuel
Aug 28
SQL Query Optimization Tips I Regret Not Knowing Earlier
SQL Query Optimization Tips I Regret Not Knowing Earlier
15 Tricks to Level-Up Your Data Game
Nnamdi Samuel
Aug 26
Mastering SQL: How to Find and Remove Duplicate Rows Efficiently
Mastering SQL: How to Find and Remove Duplicate Rows Efficiently
Optimize your SQL skills with these strategies for managing and removing duplicate data in large databases.
RAKESH CHANDA
Aug 24
7 Surprising Habits That Make Data Engineers Unstoppable
7 Surprising Habits That Make Data Engineers Unstoppable
How to Thrive in Data Engineering
Tim Webster
Aug 23
Bing New Search — End-to-End Azure Data Engineering Project using Microsoft Fabric.
Bing New Search — End-to-End Azure Data Engineering Project using M...
This project uses Bing News API within Azure to create an end-to-end Data Engineering project using Microsoft Fabric to determine the…
Musili Adebayo
Aug 20
Data Warehouse vs Data Lake vs Data Lakehouse | The Quick Guide
Data Warehouse vs Data Lake vs Data Lakehouse | The Quick Guide
Key differences to know
Humberto López-Zavaleta
Aug 16
Stop Trying to Be a Data Hero (Here’s Why)
Stop Trying to Be a Data Hero (Here’s Why)
Pitfalls of Data Engineering Heroics
Tim Webster
Aug 15
Understanding Apache Spark: A Deep Dive into Big Data Processing
Understanding Apache Spark: A Deep Dive into Big Data Processing
Learn why Spark is the go-to choice for data engineers and data scientists.
Rui Carvalho
Aug 12
How Data Engineers Win (Discipline and Fundamentals)
How Data Engineers Win (Discipline and Fundamentals)
The Real Things Every Data Engineer Should Care About
Tim Webster
Aug 8
How to Clean Up a CSV File
How to Clean Up a CSV File
Complete simple tasks faster
Violeta Rakonjac
Aug 8
What They Don’t Tell You About Being a Data Engineer
What They Don’t Tell You About Being a Data Engineer
What No One Prepares You For
Tim Webster
Aug 1
Microsoft Fabric End-to-End Project — with Shorcut, Data Pipeline, DataFlow, and Regular File…
Microsoft Fabric End-to-End Project — with Shorcut, Data Pipeline,...
The aim of this project is Microsoft Fabric Project is to:
Musili Adebayo
Jul 30
5 Lessons from 5 Data Teams (Lessons for Data Engineers)
5 Lessons from 5 Data Teams (Lessons for Data Engineers)
Common Sense Data Engineering Lessons Across Teams
Tim Webster
Jul 26
Learn with me: Using .env file in python
Learn with me: Using .env file in python
Hello everyone,
Violeta Rakonjac
Jul 25
Steady as She Flows: Rate Limiting in Apache Beam Pipelines
Steady as She Flows: Rate Limiting in Apache Beam Pipelines
A Data Engineer’s Toolkit for Controlled Streaming
Josi Aranda
Jul 21
Detecting Risks in Claims Data Using Databricks and PySpark.
Detecting Risks in Claims Data Using Databricks and PySpark.
Databricks and PySpark for Efficient Claims Risk Management in healthcare claims.
Brahmareddy, The Data Engineer.
Jul 21
Data Architecture Principles Every Data Engineer Should Know
Data Architecture Principles Every Data Engineer Should Know
Discover the key principles of data architecture critical for any data engineer.
Rui Carvalho
Jul 18
The Reality of Mistakes in Data Engineering (Tips from the Trenches)
The Reality of Mistakes in Data Engineering (Tips from the Trenches)
How to Avoid the What-the-Hell-Have-I-Just-Done Moments
Tim Webster
Jul 18
How to Fix an Error in the DBT Command with the Empty Flag Due to _TABLE_SUFFIX or _PARTITIONTIME
How to Fix an Error in the DBT Command with the Empty Flag Due to _...
Utilize dbt variables to fix the problem
Fumiaki Kobayashi
Jul 18
Data Transfer Metrics: Bandwidth, Throughput and beyond
Data Transfer Metrics: Bandwidth, Throughput and beyond
A Data Engineer’s Guide to Understanding and Leveraging Data Transfer Metrics
Santosh Joshi
Jul 13
A Data Engineer’s Biggest Hurdle (Figuring Out What People Want)
A Data Engineer’s Biggest Hurdle (Figuring Out What People Want)
Bring Real Value by Asking the Right Questions
Tim Webster
Jul 10
11 Useful BigQuery Tricks (for Newbies)
11 Useful BigQuery Tricks (for Newbies)
Tricks Every New BigQuery User Should Know
Tim Webster
Jul 1
About Art of Data Engineering
Latest Stories
Archive
About Medium
Terms
Privacy
Teams