Extract. Transform. Read.

Why Data Engineering Outlived A $200k AI Role

10-year job market data demonstrates how data engineering can outlast “trendy” tech jobs.

The following short read is an excerpt from my weekly newsletter, Extract. Transform. Read. sent to nearly 3,000 aspiring data professionals. If you enjoy this snippet, you can sign up and receive your free project ideation guide.

Remember when everyone was scrambling to become a “prompt engineer” or “metaverse architect?” Last week The Wall Street Journal published an article (The Hottest AI Job of 2023 Is Already Obsolete) exposing how the “hottest” AI job, the prompt engineer, quietly faded.

It’s a familiar and eye roll-inducing story. The tech industry has been guilty of heralding the “decade’s next job” prematurely. Looking at you, Blockchain devs, NFT traders and “Chief Metaverse Officers.” These roles often surge in popularity, driven by hype and venture capital, only to fade as the technology matures or the market shifts.

But among the volatility, one role has consistently remained in high demand (and projected to grow more!): Data engineering. The U.S. Bureau of Labor and Statistics forecasts an 8% growth for data engineering-type roles within the next decade*, which is 5% higher than the average growth rate for other professional roles. It helps that the data and analytics industry is projected to grow by over 30% by 2033, reaching a market cap of nearly 900 billion by 2032.

In true data engineering fashion, the data doesn’t lie; while it can be tempting to lump data engineering in with the “hype” that accompanied data science roles in the 2010s, roles supporting and scaling data infrastructure are more necessary than ever. Anecdotally, my team doubled in size within the last quarter. As it looks like we’re increasingly approaching the crest of an AI bubble, companies are shying away from investment in “trendy” tech and working to wring more value out of the data they already possess by doubling down on products that make that data available to internal data consumers.

Although it’s true that the tech job market overall has cooled within the last 1–2 years, there is still room for passionate, specialized individuals who can upskill and brave confusing API documentation to aggregate and extract structured and unstructured data from multiple sources.

Even with the role’s growth, it’s important to go beyond just learning Python and SQL; baseline competency isn’t enough anymore. To succeed in an increasingly competitive role, you need to cultivate genuine interest, as I noted in 2022: “[I]f you don’t enjoy learning and employing new data engineering techniques and, just as importantly, don’t admire the product and teams you are supporting, it won’t matter how much money you make because you likely won’t be professionally or personally fulfilled.”

And if you’ve mustered the passion but don’t quite know where to start, I wrote a two-part series on breaking into the data science/data analytics field in its current iteration.

Note that none of that advice includes learning how to write prompts.

*The U.S. BLS figure pertains to the data architect role, a stand-in for data engineering since BLS does not refer to DE explicitly

Thanks for ingesting,

-Zach Quinn