On Data Science Job Scarcity
Various “authorities” forecast an extreme shortage in data science workers, oh excuse me, data scientists in the near future as virtually every business becomes interested in learning from the data it collects. This scarcity is allegedly a result of the abundance (and exponential growth) of data science jobs and the lack of skilled data workers. I suspect that the actual number of data science jobs is grossly overestimated. This is because not every company has data that are data science ready. Such data either do not measure anything interesting or are too complex for traditional off-the-shelf methods to analyze. Unfortunately, if we believe these projections, we’ll train an army of people for jobs that do not exist.
If we remove jobs where adequate data are not available or jobs where state-of-the-art methods are not suitable, we are left mainly with Internet companies and large-scale hardware companies that have access to tremendous amounts of data to make statistical analysis feasible and have the ability to tweak their data-gathering efforts by merely changing a few lines of code without service interuption.
So before you spend your savings on that Data Science masters degree, you should be aware that data science in large Internet companies is narrowly defined (think more data and less science). As a “data scientist” you will be mainly working on traditional Internet company problems: how to sell more ads, better recommendations, and better image tags/labeling. Data science is an evolving and growing field. However, data science jobs are only abundent for a minute application of this broad discipline. If you really want to be a hot commodity, become a data scientist who develops novel methods to analyze complex (non-Internet centric) data.
There is this scene in the show Damages (fast foward to ~8:00): an experienced lawyer, makes an aspiring attorney sign a blank card and then scribbles “I was warned”. So were you, aspiring data scientist.
If you found this story useful, please consider recommending it and/or sharing it on social media using the buttons below.
