5 Characteristics of a Great Data Engineer

Building a team of great data engineers requires analyzing not only a candidate’s technical knowledge, but also a host of other characteristics, including analytical/critical thinking, communication, problem-solving and perseverance, attention to detail, and more.

In today’s world of rapid technological change and overwhelming choice in solutions and services, companies need to recruit data engineers who, above all, can help the company’s data platforms be more adaptable, integrated, scalable, and secure. But what are the characteristics of a data engineer who can help you do these things?

1. Be Adaptable and Curious

Due to the rapid pace of innovation and low barriers to entry, the data engineering space comprises a staggering range of competitors, from niche players to enterprise platform vendors. As a result, many technology platforms simply haven’t existed long enough or seen the widespread adoption required to form a critical mass of skilled experts. So if you can’t hire an expert, what’s the next best thing? Hiring people who are good at becoming experts!

Great engineers love to learn. This applies to learning new technologies as well as learning new areas of the company’s business. The opportunity to explore areas of passion and interest through continuing education and self-study helps data engineers keep a healthy academic interest in the profession and helps to keep up with changes in the industry as they develop. I would go so far as to argue that great data engineers need to be curious and enjoy learning new things, usually under pressure. They take almost nothing for granted and enjoy questioning long-held assumptions.

2. Have Depth AND Breadth

It is often said that software engineers require “T-shaped” skills: depth of skills in one or two areas, coupled with a breadth of knowledge across the field. With agile development and DevOps catching on within IT and product teams at most large enterprises, developers are increasingly asked to take on a broader range of technologies and responsibilities that might have typically fallen to DBAs or system admins. Developers who are deep in one area and unwilling to expand their horizons are sure to face challenges in today’s job market.

3. Take Ownership

Data engineers are not in the business of finding problems and complaining about them (though they are quite good at it), they are in the business of solving them. They take ownership and drive issues to resolution, collaborating with whomever they need to along the way. In the new DevOps world, there might not be a DBA or a system admin to help you. Empowered teams taking ownership of their own future is one of the reasons agile software development works so well.

4. Have Strong Opinions, Weakly Held

Having opinions requires conviction and sticking your neck out. Admitting we are wrong is hard on our egos. But great data engineers can do both. They can argue convincingly for the opinions they hold, but are always willing for that opinion to be replaced by a better one. The human brain is imperfect and subject to cognitive dissonance and bias, so we must constantly look for these behaviors in ourselves and remind ourselves to be open to new ideas that might break our current modes of thinking.

5. Embrace Modern Data Architecture

Moving data workloads to the cloud and to modern data platforms like Hadoop and Spark really does require a shift in thinking for many experienced data engineers. A fundamental understanding of distributed computing is a prerequisite in this space. Volume, variety, and velocity can all be tamed with the right set of tools, but it takes engineers who ultimately understand which of those they are trying to optimize for and how the technology platforms they are using fundamentally solve those problems. Scalability is paramount in a world where value can be extracted from systematically tracking down needles in haystacks, so developers who can expand their horizons and incorporate modern data architecture into their enterprise ecosystems will be ahead of the game.