AI Product Management P4 and P5: How do you know if your AI product is ready to be released?

Stella Liu
4 min readOct 27, 2019

Learn best practices on how to build your AI model and when to know when it is “good enough.”

This article is part of a series that breaks down AI product management into 5 distinct phases. The introduction to these series starts here.

AI Product Management Phase 4: Building the AI Model

Consideration: Who should be involved?

Short answer:

You’ll need to think beyond data scientists.

Long answer:

You’ll need to partner with subject matter experts, end-users and buyers. They can help you confirm that the AI model’s output is relevant or valuable to them.

1. Subject matter experts

During the training process, if you’re doing supervised learning, you’ll have to label the data. SMEs can help label the data, especially if the labeling requires domain expertise. You may consider outsourcing the labeling effort that doesn’t need SME input to save costs. For example, labeling the ‘noun’ in a sentence doesn’t need special knowledge. There are vendors out there that can do that for you like Amazon Turks or Figure 8. But, if you want to label “risk” in a legal document, you will need a SME like a lawyer to label that. After training the AI model is complete, SMEs can then help confirm the output of the AI model. For “unsupervised” learning, you won’t need to label the data since the AI model will find the trends itself. But, SMEs are still important because they can sanity check the output of the AI model.

*The word “label” simplifies the language here. Depending on the algorithm you use, your data scientist may use different terminology. But it’s the same concept — you need to “label” or “identify” the entities you want to extract from your data set.

2. End users

You should also make sure to confirm the AI model output with your end-users. In our case, after we did a training cycle, we built a prototype UI that displayed the AI model’s output in a more consumable fashion for the end-user. Instead of a JSON file detailing the AI model results, we created a prototype UI that said we found “x” in your data. We then had the end-users let us know if the recommendations are making any sense or are valuable or not. You’ll want to start prototyping early on how to display the AI model output results to the end-user. Recruit those end-users early on as you build the AI model.

3. Buyers

You’ll need the results to be useful on the individual level, but it is also critical that the results are valuable at scale. In my case, we not only displayed recommendations for the end-user, but we also enabled the buyer to create trend reports on the business. These reports showed that a specific business KPI improved by x% with the result of our AI product.

Overall Tips

  • You’ll need to involve SMEs, end-users and buyers.
  • The AI model output needs to be understandable in order for it to be valuable.

AI Product Management Phase 5: Test and Deploy

Consideration: When is the AI model good enough?

Short answer:

Never ;)

Long answer:

It’s not a 1-time training cycle. You’ll need to update your AI models so plan for that in advance. When data in production changes, your AI model also needs to adapt and change.

You can plan for this by building a continuous feedback loop in the product itself. This will allow you to track and find out when results are losing relevance for your end-users. It can be as simple as a ‘thumbs up’ or ‘thumbs down’ icon. You can also do periodic focus groups with your end-users and/or do your own internal testing with SMEs.

Additionally, you can create a “Model Ops” data science team. This team can review incoming ‘defects’ related to the AI model, triage them and then update the model. As the team gets better and better, the team can deliver updates in real-time. The last thing you want is an end-user to file a support ticket saying that the results are wrong and then have that person wait for months before it improves. With a model ops team, you can shrink that reaction time to days.

The feedback you receive here may take you all the way back to phase 1, challenging your basic assumptions and causing new AI model work.

Overall Tips

  • It’s not a 1-time training cycle.
  • When the data changes, you’ll have to change your AI model.
  • Ongoing maintenance takes time and effort.
  • Consider a model ops team.

Stella Liu was a Product Manager at IBM Watson IoT where she helped build her team’s first AI-based product at scale. She loves to talk about AI, product management and environmental sustainability.

Please reach out to her at LinkedIn for any questions or comments!

--

--

Stella Liu

I like to demystify complex tech. PM @ VC-backed Indigo Agriculture. Former IBM Watson PM & Fulbright Scholar. https://www.linkedin.com/in/stellaliu93/