Has Generative AI Killed the Data Scientist?
10 years ago, becoming a data scientist was considered the job of the century.
With the consolidation of the Web-2 network, users were not only data consumers but data producers.
The need to manage, process and interpret this new form of gold was paramount and as such, hiring Data Scientists became a high priority for any company that consumed large amounts of data.
These large amounts of user behaviours, network structures, content, demographics, psychographics and much more allowed companies to understand and predict human behaviour at levels never seen before. Data Scientists were at the front of the data stream process that allowed us to bring “life” to data.
By “life”, I am metaphorically referring to Generative AI, as the ultimate emulation of human behaviour that AI has achieved.
However, curiously, as in Mary Shelley’s Frankenstein, the newly born creature, involuntary kills its creator.
Between 2022 and 2023 over 500,000 layoffs occurred across all tech jobs globally, with over 30% of those layoffs affecting engineering and data science roles. This is not solely caused by high interest rates and their impact on the tech industry but also by the necessity for specialisation in this area, due to the rapid evolution of the available AI technologies.
What Happened to the Role of the Data Scientist?
Typically, the role of the data scientist primarily revolved around extracting insights from data, building predictive models, and optimising business processes using statistical analysis and machine learning algorithms.
This involved an initial learning curve into the technicalities of predictive analysis and the Extract Transform Load (ETL) process. Statistics, Maths, Programming, Machine Learning understanding, Data Engineering and Visualisation were the key skills for a successful career in Data Science.
However, with the widespread adoption of generative AI — most of these tasks can now be easily implemented into new levels of abstraction, with less code and a seamless infrastructure.
Tools, like Pecan, Altair and Rivery, very easily streamline the whole process of data into comprehensible business insights. This makes the technical aspect of the Data Scientist less relevant than intuition and business vision.
Big tech companies like Google, Microsoft and Amazon make it very easy for anyone with a superficial knowledge of programming and statistics to deliver powerful analysis that 10 years ago would require a deeper knowledge. This is a paradigm shift for those invested in such a career path.
Pronouncing the Data Scientists as dead might be a stretch, but certainly, a lot will change.
Let’s explore what the career landscape is like for them moving forward.
A Smaller Middle Ground — Specialisation
Typically, the data scientist was between a business analyst and a machine-learning/AI engineer. With some knowledge of ML algorithms and AI principles, a Data Scientist would trigger these methods to output insights for the business strategy.
However, this superficial knowledge can now be much more easily supported by GenAI. This leaves the Data Scientist to decide on a new path to stay relevant within an organisation.
The original middle ground position between a very technical team(ML/AI/Data engineering)and a not-so-technical one (Business) will require way fewer people. Therefore, the new data scientist will have to revert to either one of the sides of the scale.
The AI Data Scientist
A logical choice for a reinvention of the role is the specialisation in the technicalities of model creation and model testing.
Machine learning, Data Engineering and AI development will have a strong demand over the coming years. This will most probably be a job more confined to companies specialised in AI and not so much to classical industry companies.
A re-skilling into machine learning principles and methodologies, machine learning ops, data engineering and specific AI fields such as computer vision, NLP, deep learning and so on.
This is a path that will require the capacity to re-skill in a highly theoretical and technical field. It will require a mix of computer science and maths. Moreover, it might turn out to be very difficult if the original data scientific background is grounded on a more superficial understanding of statistics and functional programming.
The Data Strategist
On the other side, an alternative path for the reinvention of the Data Scientist role is more focused on data strategy and its applications to a company’s operation.
This new data scientist would be more akin to a data enabler. They would envision the whole chain of data acquisition potential within a company’s operations.
They would ensure that this is applied across the whole infrastructure to serve the company’s purposes. With a very strong focus on evangelization, this person would enable all employees to work with data, acting as data translators to the overall company data strategy.
Companies now don’t solely act on classical user data, which more or less comes in a regular way, but also on their own data to gather insights into operation improvement.
Concepts like Process Mining are becoming more popular, where internal processes are converted into data points that can then generate insights.
The Data-Driven Employee
As generative AI gets more and more ingrained within business operations, data becomes the new fuel for performance.
This means that data can no longer be seen as the property of a small team but as a commodity used by everyone in the company. Hence, Data Science will have to become a company-wide discipline.
New AI models will allow us to transform data that previously would be hard to evaluate given its qualitative nature, into a comprehensive categorisation/evaluation with very little effort.
Basically, we can now harvest almost all types of data (events, content, audio, image and video, etc) to produce insights.
Each employee will not only be a data consumer but also an avid data producer — and as such — it will have to be part of their responsibility to curate their own data pipeline to its full potential.
The Impact on Future Product Development
In the evolving landscape of business operations, every product is now compelled to integrate data considerations beyond the classical event-based Key Performance Indicators (KPIs).
No longer can product development solely rely on conventional metrics; instead, they must delve into the intricate layers of user behaviour, preferences, and feedback to stay competitive.
For instance, it may not be enough for an e-commerce platform to analyse the number of purchases alone. Rather, they will need to also look at browsing patterns, wishlists, and even sentiment analysis from customer reviews to optimise its offerings.
The new AI data scientist emerges as a pivotal figure in supporting product teams to envision these core layers.
With their expertise in machine learning, data engineering, and AI development, they can guide product development by leveraging advanced analytics to understand user needs and predict market trends.
By harnessing the power of generative AI and data-driven insights, product teams can iterate faster, tailor offerings to individual preferences, and ultimately enhance user experiences.
Companies poised for success are those that can adeptly collect, manage, and profit from their own generated data.
With the aid of generative AI-driven analytics and the strategic guidance of data experts, product development can transcend traditional boundaries, leading to more innovative and competitive offerings.
Conclusion
The trajectory of the Data Scientist has undergone a profound shift, evolving from a mere data insight generator to a crucial enabler of organizational success in the data-driven era.
As the demand for data-driven decision-making permeates every aspect of business, Data Scientists must choose between two key options.
- Embracing a more theoretical role
- Adopting a strategic stance in delivering value to a workforce increasingly versed in data literacy.
While the traditional notion of the Data Scientist may seem obsolete, it’s only set to resurface within a different data paradigm.
Armed with new skills and perspectives, these professionals stand at the forefront of driving innovation, shaping strategies, and empowering organizations to thrive in the data-centric future.
