A Practical Guide to AI Product Management: Part 1
A vast number of software applications ranging from YouTube’s recommendation algorithm to Apple’s Face ID use Machine Learning (the methodology behind implementing AI) behind the scenes. Adoption of AI in every single industry is increasing so rapidly that AI is forecasted to contribute about $15.7 trillion to the world economy and boost local economies by 26% by 2030.
So how do you, a Product Manager, get in on the AI action?
Artificial Intelligence Product Management is the practice or, dare I say, the art of building, shipping, and maintaining products in which machine learning drives key user outcomes. However, building AI-powered products is not as straightforward as building non-AI (let’s call these traditional) software products. It requires a fundamentally different strategy because of the nature of machine learning. This article covers the foundations you need before getting started. If you’re already familiar with ML basics, skip ahead to Part 2.
Knowing how to incorporate ML driven features into products is a very valuable and soon-to-be critical skill for PMs. A popular method of learning ML today is through online courses. While they will teach you how to train models and generate predictions, that’s only a small part of the puzzle.
Building models is the easiest part of implementing machine learning in your product. Everything that comes before and after is hard and not covered in most ML curriculum.
AI implementation today, despite all the buzz, has a high failure rate. You can hire the best machine learning engineers but without clarity on business objectives and the right implementation strategy, any attempt is bound to fail.
A survey carried out by Boston Consulting Group and MIT polled almost 2,500 bosses and found that 7/10 said their AI projects had generated little impact so far. Two-fifths of those with “significant investments” in AI had yet to report any benefits at all. — Businesses are finding AI hard to adopt, The Economist
One reason behind this is that business leaders and product managers don’t always know how to approach AI. It’s a new field, it’s difficult to implement and it can get very expensive if not done correctly.
This series of articles is a step by step playbook about laying the groundwork for machine learning, integrating it with your application, deploying it all to production, and maintaining it once it’s shipped.
The series is divided into 3 parts -
Part 1: Groundwork — what makes AI product management different, should you use ML for your project, prerequisite knowledge for AI PMs
Part 2: AI team management, product planning and development strategy
Part 3: Model selection, deploying to production, model maintenance and cost management
There are numerous online resources that can teach you the mathematics and programming behind training a model. We’re going to focus specifically on a PM’s perspective of how to incorporate ML in a product. It is assumed that you are familiar with established product management and Agile principles, therefore these concepts will be skipped as well.
An AI PM needs to understand how ML works. The infographic above describes how a model is built by a data scientist. Let’s look at an analogy about how it would work in production.
Consider a football/soccer goalkeeper practising before a match. Learning how to block a variety of shots through repetition is like a neural network being trained (neural networks are modelled after the human brain). Now in a production environment, which is during a match, the goalkeeper is not going to face a shot on goal that has the same speed, trajectory, bounce, spin, etc as faced during training. There’s definitely going to be some real world variation. If the goalkeeper has practised or trained his neural network enough on similar shots (similar data), there’s a high probability that he can block the shot successfully and clear the ball from his side. The key phrase here is high probability. But say the goalkeeper faces a shot with some variation in angle and spin as compared to the practice sessions. Then the probability of success reduces meaning he might block the shot but not clear the ball very well. If he faces a shot which is very different from practice then the probability of success is very low and a goal is scored. As the variation in the shots (input data) increases, the lower the probability of the goalkeeper’s success because he has not practised on those kinds of shots. To get better the goalkeeper will have to practice more or in other words, train on additional data. So it follows that the best goalkeeper in the world is the one who trains on all kinds of shots, or the most diverse dataset.
What’s the difference between AI and traditional software development?
The biggest difference is that machine learning delivers uncertain outcomes while traditional software engineering delivers objective, or deterministic outcomes. In an e-commerce website, the code for what happens when a user clicks the “Add to cart” button consistently works exactly as intended. With an ML model, the outcomes are probabilistic. The model returns an answer with a certain confidence or accuracy score (depending on the type of model). Do you want uncertain outcomes in a user-facing production application? Only if using ML has a massive advantage over deterministic software code. This brings us to the next question.
Should you even be using machine learning for your problem?
Knowing when and where to use machine learning is essential for an AI PM. B2C customers generally don’t care what you do as long as your product works. However, B2B customers are sometimes very enthusiastic about applying ML to their problems, even when it’s not needed, because they may lack the necessary knowledge. Don’t try to force-fit ML on a problem that doesn’t require it, no matter how much stakeholders insist. One of the AI PM’s most important jobs is to keep the stakeholders well informed about what you’re doing, potential outcomes and the risks involved. Be upfront about the process, data, budget and team required for this in the early stages of planning. Many will expect that just like traditional software, ML should work consistently with a 100% accurate output every single time no matter what data is input. The concept of deterministic vs probabilistic outcomes should be made clear to all involved.
Knowing what’s possible with the current state of the art ML models and what’s not is also important. This will help you tackle unrealistic expectations from stakeholders.
Here’s a helpful framework for evaluating whether ML is the right solution for your problem -
In some cases, you understandably want a 100% deterministic outcome because it can have huge monetary or health implications. So making life and death healthcare decisions solely with AI without a human expert in the loop is not a good idea just yet.
What do you need to know as an AI product manager?
Apart from general product management knowledge, a specialist AI PM needs to have a working knowledge of machine learning development and deployment. If you’re hands-on with ML, it’s a huge advantage. But to what extent you need to be hands-on depends on the problem and team composition. Preferably PMs should hover at helicopter height instead of writing code themselves because they’ve got so many different people and functions to coordinate.
Know your data like your life depends on it (your ML product certainly does)
AI PMs must know all the intricate details about the data they’re going to work with. Incomplete, unrefined or corrupted data will certainly lead to sub-par ML performance. Data management is covered in-depth in Part 2.
Keep up with the latest developments in AI. Knowing which technology is mature and what is on the horizon is extremely helpful. It takes just 1 new paper to turn what’s impossible today into a reality tomorrow and the industry is moving at a rapid pace. The best way to do this is to constantly read relevant content and follow industry leaders.
Finally, I can’t stress enough how important it is to know the basics of cloud infrastructure. Learn the cloud platform most relevant to your work. Once you learn one, it’s not too hard to work with another.
Here are some useful resources to help you get started:
- Glossary of machine learning terms
- Machine Learning A-Z Hands On Course on Udemy
- Introduction to Machine Learning Course on Udacity
- 2 Minute Papers YouTube Channel — highly recommended for keeping up with the latest developments in AI
- Steve Nouri on LinkedIn — also highly recommended for latest developments in AI
- Elvis S on Twitter for ML learning resources
- Basics of AWS
- Docker — a software containerization tool that’s very useful for ML deployment
- Kubernetes — a container orchestration tool
- ML Ops — you have a trained ML model. Now what?
Proof of concept
Starting by building a Proof Of Concept (POC) ML model is highly recommended, especially for complex applications like computer vision and NLP. Train and test your model on a subset of the data to understand whether ML can indeed be used to deliver the desired business outcomes. It also helps you figure out initial performance benchmarks, timelines and cost estimations. Work with a small team and don’t focus on data pipelines and deployment just yet. A POC also gives stakeholders the confidence to give you the team and budget for implementing this in production. If the POC does not deliver a favourable outcome, it is still a success because you will be saved from spending a lot of time, money and energy on a project that is unlikely to work.
Head over to Part 2, where we deep dive into the product planning and development process.
Part 3 covers deploying and maintaining the models in production.