The Growing DSA Divide
The Feedback Systems Are Broken…
Depending on who you believe, the United States is going to have a shortfall of half a million data science and analytic (DSA) professionals in just a few years. If the logic of that sentence bothers you, you might have a chance of being someone who takes advantage of that demand. Interestingly enough, it doesn’t seem to matter who you believe — all the predictions have the same shortfall (on vastly different definitions and denominators).
In reality, the story is far more bleak. If you talk to enough analytic leaders, which I do, you will learn that hiring is slow. Retention is impossible. But worse still, most hires are vastly under qualified. And many, many managers are afraid to fire people they know they can’t replace. If the logic of that bothers you, you might have a chance of being someone who takes advantage of that demand.
Only, it won’t last… Demand can not continue when the product is inconsistent, slow, and incompetent. Granted these are the same people often called on to measure that, but we digress. This divide is going to close, but likely not in a way that most of us would hope.
Giggling in the wings are software companies. In their minds, they hold the answer. At least, that is how they sell. You can’t blame them. They recognize the issue. It isn’t a nail… but they only have a hammer. The fact that their answer is the equivalent of offering more power tools to a world starved for trained carpenters… but then that is the world’s problem, their P&L is just fine. For now…
Analytic consultants aren’t helping. But then, you can’t blame them either. The world is telling them to sell the hole, not the drill. They certainly aren’t in the business of teaching the skills necessary to determine where the hole should be and what size. Well, at least most of them.
Consultants are busy exploiting the divide, not closing it. The wider it gets, the greater the demand for their services. They can drill a thousand holes of a hundred different sizes. If in the end, they have generated very little value, who’s to care? Likely no one, there is always something to blame. It is the nature of the divide. And their P&L… is just fine.
I suppose we can look for hope from our soon-to-be AI overlords. Only Artificial Intelligence doesn’t really thrive in the void either. AI is a tool for well-defined problems in well-defined places, with well-defined data. Sure driving a car can be a challenge, but Google and others have spent millions defining our roadway data. Traffic flows under clear rules, roadways are clearly delineated, and everything moves in just two dimensions. Most of analytics is not so well-defined.
Universities, venture capitalists, government agencies, MOOCs, and third party education systems all dance around it. The divide provides all of them opportunity, desperation often leads to rich opportunities. None of them is in a position to do much to close this gap. None of them are hands-on. They are not applied. They aren’t on the front line.
Businesses aren’t likely to overcome this on their own. This isn’t what they do. Yes, they are on the front lines but their focus is elsewhere. They are not positioned to close the divide, no matter how much of a problem it creates for them. For the most part, they are all in this together. So as long as their competitors don’t find an edge, their P&Ls will be just fine.
Once again, the thing that is going to suffer here is DSA itself. Decision Science and Analytics has become too hard, too costly, too inefficient, and too often practiced by those who are under qualified and overwhelmed. Executives are becoming desperate, confused, and ultimately jaded. This will act to deter interest and lower demand. If nothing comes to close the divide, it won’t go away on its own. It will simply fade from interest.
I have my own theories on how to solve the divide. You can read more in my other articles here on Corsair’s Publishing. Feel free to leave your own comments below. And thanks for reading!
