Member-only story

2024 In 12 Data Engineering Errors That Ultimately Advanced My Skills

My 3rd year of cataloguing errors encountered when creating data engineering pipelines in Python, SQL & Airflow.

Zach Quinn
Pipeline: Your Data Engineering Resource
9 min readDec 17, 2024

--

In what has become an annual tradition within my personal blogosphere, I now offer to you, in the spirit of vulnerability and learning, the most memorable errors I generated in 2024.

To better understand this exercise, read dispatches from the past two years:

Before proceeding, I think it is important to note that as I’ve gained experience as a data engineer and within the technical field in general, I no longer perceive errors as doomsday scenarios. Having written and deployed thousands of lines of code, I’ve run into so many recurring issues that I (almost) don’t have to read the logs.

This is especially true with pipelines I design and build. Instead of reacting like I would to a code red level disaster, errors are now, for the most part, met with mild annoyance. Barring a few specific scenarios, nearly anything can be isolated, troubleshot and, worst comes to worst, reverted.

If you’re someone working toward gaining experience in a technical field, my best piece of advice for addressing errors, aside from not breaking your stuff, is to approach them with less gravity and put less pressure on yourself. These feelings should be reserved for true…

--

--

Pipeline: Your Data Engineering Resource
Pipeline: Your Data Engineering Resource

Published in Pipeline: Your Data Engineering Resource

Your one-stop-shop to learn data engineering fundamentals, absorb career advice and get inspired by creative data-driven projects — all with the goal of helping you gain the proficiency and confidence to land your first job.

Zach Quinn
Zach Quinn

Written by Zach Quinn

Journalist—>Sr. Data Engineer; new stories weekly.

No responses yet