A miniguide to AI Product Management

This article was co-written with Matthew B.

Introduction

Working with MLOps processes can be a significant transition for aspiring AI product managers. Regardless of their previous roles as data scientists, engineers, or ML engineers, the focus shifts to managing processes, resources, and team collaboration.

This article offers guidance for aspiring AI product managers. To enhance the effectiveness of their work, we’ll try to emphasize the importance of understanding MLOps processes. Understanding continuous integration, continuous development, and continuous (re)training, is particularly important. But what might be even more important is to have a good overall process design from Ideation to Deployment with rigorous Testing to follow.

Alignment with the business

A lot has been written on this, so we’ll try to keep this part short. But of course, an effective approach for the AI product manager is to gather the right requirements from business stakeholders. This involves a strategic and collaborative process. Product managers can start by understanding business objectives and educating stakeholders about the capabilities of AI. It’s important to quickly identify key stakeholders and organize workshops to encourage open-ended discussions, and to ask questions that elicit insights and expectations.

All kinds of tools can be used: user personas, prototypes, proofs of concept, and visual demonstrations. These should help in gathering the requirements and at the same time demonstrate the AI solution’s potential. Using real-world examples always helps to build confidence and context, make it concrete! Collaboratively defining key performance indicators (KPIs) and establishing a feedback loop for ongoing involvement is also an important part of this work. Documenting and prioritizing requirements based on business value and technical feasibility is a must.

Embracing an iterative approach to accommodate evolving insights and shifting priorities is what you want. Managing expectations from the business that continued time investment on their side is needed must be clear.

Iteration is king.

Additionally, proactively addressing concerns about AI, including ethical considerations, model governance, and data privacy shouldn’t be forgotten.

Data first approach

As tech leaps forward, the spotlight’s back on data quality. And rightfully so. Model quality relies on data quality, making it crucial for AI product managers to view data as a product too ex. To establish a minimum viable AI product, there must be a minimum viable dataset. ML models are not islands. We are seeing many product managers that work in isolation with just the data science or AI team. However, understanding how the data influences the models should always be a part of the product manager’s job description. This will require talking to all key roles involved in the data processing pipeline, including data engineers, data storage managers, data scientists, and anyone who has engineered features or contributed transformational logic to the raw data.

Consistently analyzing data fed into an AI product is crucial for its success. This involves meticulous planning for model and data iterations, creating project roadmaps and risk assessments, and incorporating integration strategies, development approaches, and training methodologies. By embracing these elements, a smooth and efficient workflow is ensured, contributing to the reliability and overall success of the AI product. AI Product Managers should carefully consider the downstream effects of data changes on the model, as well as the deployment process.

Grasping automated deployment

AI product manager’s comprehension of CI, CD, and CT in the MLOps context is pivotal for coordination:

• CI (Continuous Integration): With frequent updates to the recommendation algorithms, there’s a need to ensure they are integrated seamlessly. The product manager orchestrates this by liaising with the data science team. They make sure that every model iteration is validated against performance metrics that matter both technically and to the business, ensuring the model’s efficacy remains optimal.
• CD (Continuous Deployment): The product manager collaborates with the data science teams to smoothly integrate improved models into the production environment. It’s not just about deployment, but also ensuring the models align with user needs and business goals. By leveraging MLOps practices, they automate deployment, accelerating the impact of model improvements.
• CT (Continuous Training): Market and user behaviors are ever-evolving. AI product managers must use MLOps tools to ensure the recommendation model is regularly retrained with fresh data. They bridge the gap between data scientists and the business side, facilitating the demoing of new models, retraining, and adjusting product goals to match market and user behaviors.

In this MLOps framework, the AI product manager acts as a linchpin. They are advocates and facilitators of communication and coordination. To break up bottlenecks, they must work with teams to create a robust end-to-end ML pipeline that covers data collection, model selection, deployment, monitoring, and retraining.

Being a linchpin for collaboration

A good product manager acts like a bridge, connecting different teams and making sure they understand each other. AI projects involve various people like designers, data scientists, engineers, and bosses. Product managers help teams work well together and provide the resources they need. Sometimes, despite everyone’s efforts, the work doesn’t match the business’s needs.

One common problem in any project is the difficulty of coordinating different tasks. This also applies to managing AI products. There are often delays. The solution is clear communication that everyone can understand. Product managers need to create easy ways for teams to talk and collaborate. They explain the business goals to technical teams and help business teams understand how AI works. They also track progress and ensure that AI features fit the business plans and schedules.

By taking the time to understand both product goals and user needs, product managers build trust and also help technical teams work faster and better, making sure AI products are useful and successful.

Continuous Listening

This requires hearing all parties out at all times: data producers, model developers, data consumers, and executive level. So while the tech teams should deliver solid CI, CD, and CT pipelines, we would like to introduce a new core skill for AI product managers: Continuous Listening (CL). The product manager shouldn’t be the big shot who makes all the calls by rank. Instead, they should be able to harvest all the knowledge and ideas around them, synthesize them in the right direction, and check back with all stakeholders if they are on board.

Master retraining strategies

Additionally, product managers must deeply understand the ML model retraining cycles. It is important to note that the quality of models may differ between code changes and retraining cycles due to variations in data.

For example, if a navigation app’s ML model is updated to recommend fewer turns for a smoother journey, it might inadvertently direct users to congested routes. Similarly, retraining on data specific to winter might make the app avoid areas prone to snow even during summer months.

Considering the downstream implications of data changes on the model is essential. This includes understanding how model retraining cycles could influence the quality of models, especially as data changes. Model quality can vary between code deployments and retraining sessions because of data fluctuations.

A high-quality AI model isn’t just accurate or statistically sound. It’s tailored for its purpose and supported by robust data. For AI product managers understanding implications is essential. CI, CD, and CT pipelines can keep the solution robust. However, keeping the model sharp, adapting data changes, and ensuring that the end product remains top-notch, user-centric, and valuable requires humans in the loop.

Process design

A golden nugget for product managers is having a good process in place. Here we would argue that just as data and code pass through a pipeline with all its steps, checks, and tests — so should an AI product. It is up to the product manager to ensure this process “pipeline” runs correctly. Unfortunately, it is not an automated process. A human, in this case the product manager, will have to ensure it runs.

So if it’s not automated, why then call it a pipeline? Well, this analogy can be helpful by identifying different stages and artifacts. Raphaël suggests this could look something like this here in the diagram below. Similar to an approach that he has used in his previous projects.

For each stage, it is important to exactly describe what meeting(s) should be had, who should be present, who is in the lead, who is consulted, and who is informed. Additionally, there should be templates for all Artifacts and Documentation. This will ensure smoothness and transparency for everyone, in the project management part of product management.

Conclusion

We have both done AI product management work, and we can say that being an AI product manager is not easy.

Good product managers in this field are a rare breed. They need to be knowledgeable in a lot of domains and have people skills. They don’t need to be technical experts, but they need to understand the fundamentals and processes of all components involved in building AI products.

Staying abreast of the latest AI and MLOps insights is essential to developing high-quality AI products, improving coordination between teams and stakeholders, and creating a good vision for successful AI products. Eventually, this will result in happier teams and quicker time-to-market releases.

The product manager is more than just a person calling the shots, having lots of meetings, and making slide decks. We hope to have given you a starting miniguide that goes beyond this cliché and helps you understand what entails successful product management. If you have any further questions, please reach out to us and we can chat!