Homepage
Open in app
Sign in
Get started
Making Sense of Data & Helping Others Grow: Tips, Advice, and Stories from the Front Lines of Data Engineering
MOST READ
WRITE FOR US
FEATURED ARTICLE
LinkedIn
Follow
7 Surprising Habits That Make Data Engineers Unstoppable
7 Surprising Habits That Make Data Engineers Unstoppable
How to Thrive in Data Engineering
Tim Webster
Aug 23
Bing New Search — End-to-End Azure Data Engineering Project using Microsoft Fabric.
Bing New Search — End-to-End Azure Data Engineering Project using M...
This project uses Bing News API within Azure to create an end-to-end Data Engineering project using Microsoft Fabric to determine the…
Musili Adebayo
Aug 20
Data Warehouse vs Data Lake vs Data Lakehouse | The Quick Guide
Data Warehouse vs Data Lake vs Data Lakehouse | The Quick Guide
Key differences to know
Humberto López-Zavaleta
Aug 16
Stop Trying to Be a Data Hero (Here’s Why)
Stop Trying to Be a Data Hero (Here’s Why)
Pitfalls of Data Engineering Heroics
Tim Webster
Aug 15
Understanding Apache Spark: A Deep Dive into Big Data Processing
Understanding Apache Spark: A Deep Dive into Big Data Processing
Learn why Spark is the go-to choice for data engineers and data scientists.
Rui Carvalho
Aug 12
How Data Engineers Win (Discipline and Fundamentals)
How Data Engineers Win (Discipline and Fundamentals)
The Real Things Every Data Engineer Should Care About
Tim Webster
Aug 8
How to Clean Up a CSV File
How to Clean Up a CSV File
Complete simple tasks faster
Violeta Rakonjac
Aug 8
What They Don’t Tell You About Being a Data Engineer
What They Don’t Tell You About Being a Data Engineer
What No One Prepares You For
Tim Webster
Aug 1
Microsoft Fabric End-to-End Project — with Shorcut, Data Pipeline, DataFlow, and Regular File…
Microsoft Fabric End-to-End Project — with Shorcut, Data Pipeline,...
The aim of this project is Microsoft Fabric Project is to:
Musili Adebayo
Jul 30
5 Lessons from 5 Data Teams (Lessons for Data Engineers)
5 Lessons from 5 Data Teams (Lessons for Data Engineers)
Common Sense Data Engineering Lessons Across Teams
Tim Webster
Jul 26
Learn with me: Using .env file in python
Learn with me: Using .env file in python
Hello everyone,
Violeta Rakonjac
Jul 25
Steady as She Flows: Rate Limiting in Apache Beam Pipelines
Steady as She Flows: Rate Limiting in Apache Beam Pipelines
A Data Engineer’s Toolkit for Controlled Streaming
Josi Aranda
Jul 21
Detecting Risks in Claims Data Using Databricks and PySpark.
Detecting Risks in Claims Data Using Databricks and PySpark.
Databricks and PySpark for Efficient Claims Risk Management in healthcare claims.
Brahmareddy, The Data Engineer.
Jul 21
Data Architecture Principles Every Data Engineer Should Know
Data Architecture Principles Every Data Engineer Should Know
Discover the key principles of data architecture critical for any data engineer.
Rui Carvalho
Jul 18
The Reality of Mistakes in Data Engineering (Tips from the Trenches)
The Reality of Mistakes in Data Engineering (Tips from the Trenches)
How to Avoid the What-the-Hell-Have-I-Just-Done Moments
Tim Webster
Jul 18
How to Fix an Error in the DBT Command with the Empty Flag Due to _TABLE_SUFFIX or _PARTITIONTIME
How to Fix an Error in the DBT Command with the Empty Flag Due to _...
Utilize dbt variables to fix the problem
Fumiaki Kobayashi
Jul 18
Data Transfer Metrics: Bandwidth, Throughput and beyond
Data Transfer Metrics: Bandwidth, Throughput and beyond
A Data Engineer’s Guide to Understanding and Leveraging Data Transfer Metrics
Santosh Joshi
Jul 13
A Data Engineer’s Biggest Hurdle (Figuring Out What People Want)
A Data Engineer’s Biggest Hurdle (Figuring Out What People Want)
Bring Real Value by Asking the Right Questions
Tim Webster
Jul 10
11 Useful BigQuery Tricks (for Newbies)
11 Useful BigQuery Tricks (for Newbies)
Tricks Every New BigQuery User Should Know
Tim Webster
Jul 1
SQL Aggregation: Rollup Explained
SQL Aggregation: Rollup Explained
Understanding SQL Rollup for Enhanced Data Aggregation
Santosh Joshi
Jun 30
How Idempotent Data Pipelines Prevent Data Consistency Issues
How Idempotent Data Pipelines Prevent Data Consistency Issues
Discover the concept of idempotent data pipelines in maintaining data consistency in your database. Learn practical strategies to build…
Rui Carvalho
Jun 28
Why Data Engineers Must Balance AI Use with Hands-On Problem Solving
Why Data Engineers Must Balance AI Use with Hands-On Problem Solving
Embracing AI Without Losing Critical Skills
Tim Webster
Jun 20
Exploring the Latest Features in Databricks: A Comprehensive Guide
Exploring the Latest Features in Databricks: A Comprehensive Guide
Databricks Latest Features Explained.
Brahmareddy, The Data Engineer.
Jun 17
About Art of Data Engineering
Latest Stories
Archive
About Medium
Terms
Privacy
Teams