Top 5 Things You Need to Know as a Data Engineer
Here is a list of 5 things that you should know about to be an effective data engineer…
1. Become an Architect Nerd
I guarantee that most of you will or have worked with a Data Architect before. Even if you work at a small company with 0 architects, often the data engineer has to become the solution architect and delivery lead. Understand cloud architecture and data structures in detail so you can save the pain of designing something monolithic that you’ll have to watch it tick over all the time.
It’s a good thing Uncle Bob has us covered in Clean Architecture — take a look at the summary.
2. Identify Overkill
And more importantly, stop when you realise your solution takes 5 hours to explain to someone. Simplicity is key; why create a sophisticated process that kicks off 20 tasks to load a CSV with 10 rows? It doesn’t make sense. What it does mean is you will spend much longer developing something that doesn’t drive high value in a short time period. Plus, it’s boring.
“Big Ball of Mud” — John Spacey
Don’t touch it, or it’ll break…
