15 Key Learnings for Product Managers in ML Engagements, Part 1
Did you know that an overwhelming 35% of Amazon’s sales comes from product recommendations?
Did you know that Spotify’s Discover Weekly and Release Radar playlists are machine-generated and account for 31% of all listening on the platform?
Add to it, did you know that an impressive 80% of Netflix’s stream time is through its recommender system that translated to $1B in savings on customer acquisition?
Well, it must come to you as no surprise that these technology companies share a common trait — a strong data-driven culture that leverages the latest technologies such as AI, Machine learning and Deep Learning.
This was evident when Gartner expanded its coverage of AI technologies in the ‘Gartner Hype Cycle for Emerging Technologies, 2020’ published earlier in the year. It also cited that 80% of emerging technologies would have AI foundations by 2021.
While the top stories are exemplary, many organizations across the globe are still grappling to derive substantial business value using these disruptive technologies. According to IDC, 25% of organizations worldwide that are already using AI solutions report up to 50% failure rate. The survey also revealed that unrealistic expectations and lack of skilled people were amongst the top reasons for failure.
For the successful companies, a ‘Data First’ or ‘AI First’ culture has helped them not only to align their people and processes, but also their goals. They thrive in developing AI/ML platforms that are continuously tuned to stay efficient as business priorities change.
Publicis Sapient — Success Stories
At Publicis Sapient, product teams have helped global businesses become future-ready by accelerating the adoption of AI & ML-based technologies. We had helped an automotive client achieve 900% increase in test drives in one Asian market, sizable growth in capturing prospect information, and the ability to analyze +1K data points continuously. More on this here.
A few years back, Publicis Sapient had also engaged with a global fast food company in revamping their drive-thru digital menu boards from being just a menu board to a real-time interactive marketing channel delivering data-driven insights that in turn drove incremental revenue and improved customer satisfaction and loyalty. More on this here.
Based on our experiences, product managers need to play a pivotal role not just in orchestrating business value delivery, but also in working extensively with ML engineers, data scientists and data engineers with an appreciation for the experimental nature of ML engagements.
We are happy to share some of our key learnings from ML engagements, and we hope that product managers who are starting out their ML journeys find them useful.
Before we get to the key learnings, let’s first understand what an ML product development lifecycle is.
ML Product Development Lifecycle
A typical ML product development lifecycle outlines the major phases of the product development. It is crucial for product managers to understand their responsibilities as well as core skills required across these phases. Importantly, the cyclical nature of these phases depicts the true DNA of every ML engagement: Experimental & Continuous Improvement.
Now, let’s understand each of these phases in detail in the following sections. Each section concludes with a table that captures the key learnings as well as an indication of the level of involvement expected from a product manager in the respective phase.
Business Understanding
Understanding the business context and strategic imperatives are key to the success of your product. Product managers need to work closely with all relevant stakeholders to establish a solid understanding of the ‘what’ and ‘why ‘of the product.
Case in point: As per the global fraud report from CyberSource, a significant proportion of global eCommerce businesses allocate their annual eCommerce fraud management budget to order review staff as well as system automation. For an e-Commerce business that is growing, the reliance on review staff and/or rule-based transaction review systems can eventually pose a problem in the form of increase in false positives as the transaction volume grows. Machine Learning is a definite go-to strategy in such a scenario, given it can process large datasets and detect patterns at scale with reasonable accuracy.
As a rule of thumb, when your business goal falls into the category that requires the implementation of a complex logic, advanced personalization, faster scaling up of business processes, or real-time rule generation, then more often than not ML is a definite go-to strategy.
One other way to look at it will be how business decisions are made today. Say, if human predictions are more or less accurate and an ML model is performing at the same levels, then you are better off making decisions the human way. However, when the prediction accuracy is not marginal, and there is significant improvement while deploying an ML-based decision support system, then there is a definite business case for ML. For example, an experienced sales manager at a toy manufacturing company, in all probability, will serve as a good prediction agent for demand forecasting with +/- 30% deviation based on this experience and intuition, and such deviations will invariably translate to inventory cost buildup or lost sale revenue for the organization. An ML-based demand-forecasting system that can make predictions at deviations +/- 10% thus adding business value.
As you can imagine, even with per-unit cost of $20-$25, the impact on business performance is huge even with a slight improvement (5%, 10%, 15% …) in predictions.
For a product manager, this is one of the most important phases in the lifecycle for ensuring success.
Goal Setting
Business goals translate to product goals in this phase. Given the experimental nature of ML engagements, ML goal consists of an observable quantifiable success metric and its target value (definition of success). Common success metrics are accuracy, precision, and F1-scores (precision vs recall) to name a few. For example, the success metric could be product recommendation accuracy of your e-Commerce business or accuracy of defaulter prediction in credit business.
In situations where there are no prior performance reports available, rather than setting goals based on qualitative information, or simply heuristics, product managers need to work with ML engineers to perform an expected model performance exercise that is similar to a technical spike that can be done without spending much time and effort. This step comes in crucial in getting the initial buy-in from leadership, especially in organizations that are just starting out in their ML journey.
It is highly recommended that product managers revisit ML goals on an ongoing basis to assess how well the model is performing over time. It also serves as a feedback mechanism for revisiting business goals whenever applicable.
The diagram below represents the goal-setting process as well as the importance of validating goals in successfully delivering great products.
Case in point: In a recent engagement with an automotive client, the business objective was to bring down their product development research cycle time from 3 months to 1 month (a 66% reduction in cycle time). The ML-based product that we delivered was an article recommendation tool that will serve researchers with a contextual search feature to retrieve the most relevant articles. From a goal-setting perspective, the success metric was the % of relevant articles within top 100 listed articles and success was defined as 95% relevancy (i.e. 95 out of top 100 articles must be relevant to the topic the user has searched).
Another example could be that of a retailer who has set a target to increase its Total Sales Volume by 20%. Out of the many applicable ML-based strategies, a probable strategy could be the refinement of personalized product recommendations that will lead to more conversions. If the current success rate is 10%, then we could consider setting a target rate of 15%. Often, there is a tendency to setting a higher value, say 50% for this example, but we do not recommend it.
The goal-setting exercise also helps in assessing whether we have a valid business case for using ML or not. If the expected model performance is not in alignment with business expectation, probably ML is not the ideal go-to strategy. Say, if you are developing a weather prediction model and the best accuracy that you could achieve at the end of multiple modeling iterations stands at 70%, then it still does not meet the objective of near-precise prediction.
Authors: Nithin Subhakar & Shubham Tripathi
