Member-only story
2024 In 12 Data Engineering Errors That Ultimately Advanced My Skills
My 3rd year of cataloguing errors encountered when creating data engineering pipelines in Python, SQL & Airflow.
In what has become an annual tradition within my personal blogosphere, I now offer to you, in the spirit of vulnerability and learning, the most memorable errors I generated in 2024.
To better understand this exercise, read dispatches from the past two years:
Before proceeding, I think it is important to note that as I’ve gained experience as a data engineer and within the technical field in general, I no longer perceive errors as doomsday scenarios. Having written and deployed thousands of lines of code, I’ve run into so many recurring issues that I (almost) don’t have to read the logs.
This is especially true with pipelines I design and build. Instead of reacting like I would to a code red level disaster, errors are now, for the most part, met with mild annoyance. Barring a few specific scenarios, nearly anything can be isolated, troubleshot and, worst comes to worst, reverted.
If you’re someone working toward gaining experience in a technical field, my best piece of advice for addressing errors, aside from not breaking your stuff, is to approach them with less gravity and put less pressure on yourself. These feelings should be reserved for true…