Data Engineers/Plumbers — Superheroes in disguise

I am talking about different archetypes of data engineering and covering skills and seniority levels based on my experience.

Introduction

“Data is the new oil”, or “we’re data-driven”. There are many different sayings, and people focus on what’s more visible. In this post, I’ll talk about the folks whose work is visible, usually only when something breaks. This post will be the longest in the series of the Data Roles since this is where I have the most experience and knowledge picked up over the years. Let’s talk about data engineers.

Archetypes

Now, to systematise what I wrote on Linkedin about different Data Engineer Archetypes.

Streamers

No, not Twitch or OnlyFans, but Data Engineers. Those who have a particular interest in working mainly with streaming data. These individuals specialize in real-time data ingestion and processing. They are proficient in technologies like Apache Kafka, Apache Flink, and/or Apache Spark Streaming, which enable high-velocity data…