Sharing learnings from my experience as an AI Product Manager
In my last blog, I shared my journey to a PM for ML Services and this write up is an attempt to share my experience on the foundational non-AI factors that influence AI products.
In a world where AI remains the icing on the cake and grabs all the attention right from inception to an AI product release, but in reality, it is only a piece of the pie and not the end-to-end solution.
Working as Product manager for Advanced Analytics and Machine Learning products that enhance consumer facing experiences on our mobile and web applications, the following non-AI factors complete the AI puzzle…
- Prioritize, what matters to the Customer
“You can close more business in two months by becoming interested in other people than you can in two years by trying to get people interested in you.” ―Dale Carnegie
Every stakeholder has skin in the game and the view of an expected outcome varies. In my current role, I lead the ML product that can detect and block BOTs that jeopardize consumer experience. Well, the application teams want to keep consumers happy with a frictionless experience whereas the legal and privacy team’s priority is to ensure we can explain the detections in lieu of protecting consumer rights and they both are right from their lens.
While it is paramount to understand and listen to every customer need, it is also true that products can be successful only if we can prioritize. For any AI Product Manager an absolute and critical exercise is to keep an eye for the biggest bang for the buck and this has nothing to do with the AI solution. Through-out the life cycle of the product it is inevitable for the product manager to constantly articulate the vision to each stakeholder. There are always detractors, keep the lights on and technical debt that we have to address as a PM but ensuring that we don’t lose focus on the key value drops relies on the planning and prioritization skills of the Product manager.
2. Junk In, Junk Out — Data is the backbone
Data is the biggest chunk of the battle to release a successful AI product. Whether be an intuitive recommendation engine or a complicated anomaly detection tool, if there is bad quality training data then the AI model suffers from bad performance. Irrespective of the format of data sources, the success of an AI product manager depends on being data centric.
Data discovery, exploration, analysis and preparation are prerequisites for AI modeling. Data fluency is a strong asset for a product manager in all stages of the product life cycle. Being able to provide context to engineers, ability to monitor the expected outcomes of the product and being able to learn post-production incidents due to data quality and proactively deploy controls for identified gaps is crucial. Well, it is a different topic for debate :) but unarguably AI project success relies on the data plumping that feeds into the models though data scientist is the most sought-after job in the market today!
Unlike traditional software projects, AI projects have a vicious cycle of training, testing, experimenting and monitoring since AI products are not born intelligent and we work towards improving the performance at every step of this process. A lesson to share from a challenge I faced in AI solutions using consumer data, is building features on real-world high dimensional data sets like log data, telemetry and time-series data. They lack quality over time, are complicated and massive, lack governance and recovery of data is time and cost intensive. But as a Product Manager one has to constantly balance between the benefits to the model and prioritize data sources based on the lift to the model predictions. For example, in a fairness model that we did not add ‘X’ training data source since we knew it will barely be lifting the model’s F1 score and that helps focusing on what matters to solve the problem.
There is a constant balance between the data used for a proof-of-concept AI solution to actually being able to experiment or A/B test a modeling solution. In an effort to identify counterfeit products in the market, we had used a well governed sample data set and were able to fetch incredible outputs from the POC model. Well, in an attempt to extend the same to a production use case the model performance dipped due to lack of quality production data.
3. Platform is the crux, but dare to dream
“If you can’t fly then run, if you can’t run then walk, if you can’t walk then crawl, but whatever you do you have to keep moving forward.”
― Martin Luther King Jr.
This is a lesson to share from my experience on the journey from being able to accomplish ML solutions from disparate platforms to aspiring to have a comprehensive data science platform that will increase the squad velocity which was spent on custom configurations and the need for an end-to-end ML solution under one hood. Again, this is dependent on the AI maturity curve of the organization, but when I started out building AI products there was not a ready-to-go comprehensive data science platform for ML products and also want to stress that should not stop anyone from venturing into AI solutions.
While there were added costs and customization over head when it came to data preparation, training, inference pipeline- it was never a deterrent to accomplish these tasks without a standard ML platform. But there are two capabilities that clearly stood out and were determined to be extremely time consuming to integrate with in-house capabilities: 1) Model monitoring 2) Model Explain-ability. Model monitoring is crucial irrespective of the AI use-case and model explain-ability is extremely important for sensitive use-cases where decisions based on a black-box algorithm can be detrimental to the brand image. As a product manager, it is important to collaborate with the enterprise platform teams to ensure the right capabilities are designed to provide a conducive platform that will increase team productivity and enable AI solutions built at scale.
4. Always Articulate the End value to the stakeholders
Isn’t it a no-brainer that Product Managers have to live and breathe value of the product, it’s not AI PM specific :). What am I eluding here is to work towards demonstrating the value and articulating in the language of the customer well ahead of the life cycle of the product!
During the inception of the product to build an AI model in lieu of fraud and BOT detection, all stakeholders were well, yeah, we need AI — no doubt about it. But when it came to actual implementation there was clearly hesitation if the AI product on my sales pitch will actually deliver meaningful expected outcomes. In order to get the buy-in to build an experience fairness model, I planned for really quick, dirty, but still useful to tell the story and demonstrate the value that matters to the stakeholder. I worked with the focused group of stakeholders through-out the delivery. Partnered for a 3-week proof of concept with historic production data, developed ML algorithm, simulated outcomes of the model. As part of the demo, shared the benefits and risks associated with introducing AI and VOILA we were given approval to deploy the model in production.
While early reproducibility of the value and expected outcomes to the customer is the key, “How to Share the Value” is also equally important. Depending on the type of stakeholder the lens to expected outcomes will vary. For a highly technical stakeholder the ML performance like the F1 score/ accuracy or the operational excellence metrics may matter, but for business stakeholders and executives it needs to be translated to value drops like increased sales, revenue or the lined-up key performance indicators for the product. While operational excellence is still a key performance indicator for the solution, it is the paramount for the product manager to focus on the value proposition to the customer.
Well, I will conclude for now that these non-AI factors (but not limited to) are central to the success of an AI product: Identify what matters to the customer, be data-fluent, invest on a platform for the end-to-end solution and Hyper-focus on the value to the customer.