Member-only story
These 6 Data Engineering Shortcuts Will Burn You In Year 1
For new data engineers, it’s tempting to cut corners; avoid common development shortcuts to work toward advancement.
I received one of the most accurate and ominous pieces of advice during year 1, week 1 into my role as an entry-level data engineer.
Noticing my visible nervousness as I struggled to ingest a deluge of information on everything from basic BigQuery features to the intricacies of Airflow production DAGs, my then-senior engineer said: “You don’t have to know this all now. You might be good at this in a year.” Thanks to a slow fall (in which I could refine and master job-critical skills) and sheer will, I felt most “up to speed” in the next 3–6 months.
But then, just as I felt like I was a productive member of the team (no longer on “new guy” probation), I felt myself stumble a bit. Luckily, there was no singular, catastrophic inciting incident. Just a series of small blunders and slip ups, mostly noticed by me, that would delay my process or, in worst-case scenarios, the projected release dates of my builds.
Where I had lacked confidence before, most of my issues stemmed from being a bit overconfident in a job I really had only been doing for a few months. This overconfidence combined with an ambition to excel led me to cut corners so I could take on and complete more work.