Machine learning emergence: A case study of how algorithms became a team capability
“How do machine learning capabilities emerge in the absence of a central strategy?”. Based on my master’s thesis at the Vrije Universiteit Amsterdam, I answer this question by looking at how one Analytics team in a global organization within the household appliance industry has implemented machine learning methods, by providing a close account of their machine learning development.
Machine Learning (ML) has become prominent in today’s organizations to obtain a competitive advantage in their advanced analytics, by transitioning from descriptive to predictive and ultimately prescriptive analytics. This data strategy implies that ML should become a capability, but organizations omit how these capabilities should evolve leaving it at an abstract level. This results in a shallow central analytics strategy with much room for interpretation of ML methods and processes that may not lead to a higher level of analytics maturity. The evolution of these ML capabilities via individual analytics or Data Science teams has not been well studied and leaves room for which human, technological and organizational capabilities are needed to implement ML.
A machine learning capability evolution model
The research presents three main capabilities that have an impact on machine learning evolution: Human, Technological and Organisational.
Human capabilities: the skills, knowledge, know-how, and team resources available for the analytics team to direct ML initiatives.
Technological capabilities: The maturity and the functional capacity of the software available within the organization to perform ML methods.
Organizational capabilities: The capacity for the organization (ex. partnering stakeholders) to understand the importance of ML initiatives, the ability to understand the technical aspect of ML results, and the aptitude to implement ML results for value creation.
These three capabilities lead towards the development of ML within the organization, through a pattern of evolution. In this study case, one event triggered a change in one capability which led to the change of the other two capabilities and vice-versa, forming a positive feedback loop towards the development of ML. The initial trigger was the action of one of the ML practitioners to champion ML with stakeholders within the organization. With ML awareness being built, the organization started to understand the potential application of ML and thus created a demand for more human resources and skills for ML. This in turn led to more skills and resources to be added to the organization, through hiring or ML training. This led to a higher overall human capability for ML development.
With newly added ML resources, teams need to understand how to better use and scale their available technological landscape (usually restricted by company policies) towards ML use. In cases where the technology at hand is not sufficient to run ML models on, practitioners build their own solutions to enable ML. In one instance, the studied team lacked access to a data warehouse solution that would enable retrieving large amounts of data. The team resorted to building their own database to successfully query large volumes of data and use it towards ML, exceeding the capacity offered by the organization. This led not only to the temporary improvement of the technological capability of the company but also led to gains in the overall ML knowledge as the team had the necessary tools to put their ML ideas into practice.
A higher technological capability and know-how of the ML environment will lead ML practitioners to deliver products in a shorter time frame and more clearly to stakeholders. Better technology or better use of current technology drives higher effectiveness to deliver results to stakeholders. In the studied case, with the implementation of the Google Cloud Platform (GCP), the analytics team began to run their ML models efficiently, at a lower time cost, and deliver them directly through visualization software such as Tableau (with GCP out-of-the-box connectors). Thus, stakeholders received ML results faster and understood ML models better, resulting in a high buy-in for ML. Thus, technological improvements lead to an improved process of delivering ML results. Stakeholders will obtain more value out of the ML results which will drive additional buy-in and investment into ML resources and technology.
A self-reinforcing dynamic between human, technological and organizational capabilities can lead analytics teams to develop ML methods over larger time spans. This evolves organically as teams dedicate more resources in all three capabilities, ultimately evolving from a level where simple regression algorithms in Excel are used, towards an autoML functionality at scale. In this evolution, the impact of ML practitioners on the organization increases, as they start to hold more critical decisions in their hands and establish their role as trusted advisors.
Increasing explainability gap
As capabilities evolve, ML methods tend to become more complex. In this study, the team shifted towards more complex methods as time went on. In the span of a few years, the team evolved from no ML use to a level where they were using 500 time-series models and automatically selecting the best. The downside of this is that with the increasing complexity of ML models, the relative explainability of these ML models decreases due to their highly technical nature. This leads to stakeholders in the organization having challenges fully understanding the results delivered as they don’t understand what is going on behind the ML methods used. This can result in stakeholders not offering enough feedback towards the ML practitioners and lead to an increased time of implementation. As one ML practitioner mentioned in the study:
“We can only do the machine learning work that we can do. And if nobody understands what we’re doing or why we’re doing it, it kind of goes over people’s heads. They say it’s nice and move on”
Thus, an increasing explainability gap emerges between the ML results delivered by practitioners and the technical knowledge of stakeholders, raising challenges for successful ML implementation. The increasing complexity of ML results can exceed the capacity of stakeholders to learn and understand these results. This ML evolution becomes paradoxical as practitioners try to increase their overall ML capability through new methods but the new methods don’t match the knowledge of the stakeholders. This might result in the failure of the ML implementation and in lesser stakeholder buy-in for ML. To avoid this scenario, ML practitioners will need to make sure they build the ML knowledge of stakeholders at the same pace with the complexity of their ML models.
Insights for organizations that think about implementing ML
This research provides insight for managers of analytics and data science teams into the importance of investing their time in championing higher organizational capabilities. Before an investment is made into more human capabilities or technological capabilities, managers should assess the current level of ML understanding of stakeholders. Even though investing in new tools or resources might create a high initial buy-in for new ML projects, ultimately this could reduce the buy-in if the organization is not prepared for a more advanced level of analytics.
Second, as ML capabilities scale up, practitioners of ML need to recognize the pitfall of providing exceedingly complex ML results to partnering stakeholders. They should adjust the ML models according to their audience by finding a balance between explainability and complexity of results, using models that promote understanding. ML practitioners need to control the increase in complexity of ML models by always aligning it with the technical expertise of stakeholders.
Lastly, this research provides practical implications for analytics teams who want to advance their analytics level by using ML methods. Teams that aim to implement ML but lack a current corporate top-down data strategy that includes ML, need to take into consideration the enabling power of bottom-up innovation. By providing the freedom and space to innovate with new ML methods, teams can build this capability themselves without having to wait for an official directive for ML from upper management. This can accelerate the overall analytics development of organizations and provide an earlier competitive advantage for organizations.
Thank you for reading this exploration into ML development and if you are wondering how this might apply to your organization, feel free to reach out at info@rond.nu or at alexandru.sabau@rond.nu.
