Different Types Of “Data Engineering” Teams

The role of data engineer has morphed drastically in the last decade. A decade ago it seemed like employers thought their data scientists should both be able to calculate eigenvectors and understand how to write MapReduce jobs for Hadoop.

Eventually, this work became more specialized and the term data engineer started to appear more and more. Originally referring to those faithful data practitioners who interacted with Hadoop in its purest form. Then later being assisted by tools such as Flume, Sqoop, and Pig.

As time continued, the role of data engineer started to bifurcate. At larger organizations at least. Perhaps they merely started to fit into the old roles that used to hold their place such as ETL developer. Regardless, individuals and teams began to specialize.

Generally speaking the breakdown at varying-sized companies might look something like the diagram below.

These were four different data team set-ups based on people I talked to and companies I have worked with.

There is far from a perfect set-up when it comes to teams. However, you will want to make sure you balance the work being done, your data quality, output, security, and usability.

I have worked with companies of all sizes and seen some of the combinations above. But what…