A new day in the life of a Product Manager
or: How AI is changing product management, one prediction at a time
AI has unquestionably transformed the way companies build products. It’s not surprising that several job families are trying to adapt to it. It’s definitely not surprising that Product Managers have been among the first to jump on to the bandwagon. I should know. I am unashamedly one of them. From joining an ML-heavy product team a few years ago to validating/discarding GenAI ideas in the last year to unconsciously (sometimes consciously) bringing AI into my everyday conversations with friends & family — I’ve been trying to make sense of how this technology is changing my life, while learning and unlearning a bunch of things along the way.
But so what if AI is proving to be highly valuable? Why should the PM role change as a result? The last big shift that affected product management was arguably the introduction of the iPhone and the app store. There may have been a brief period where folks working on apps were calling themselves Mobile Product Managers, but this distinction did not live on. There may have been “Blockchain PMs” circa 2021, but that didn’t last for long either. Did AI then change PMing in a significant way? I think the answer is a resounding yes.
Here are a few things I’ve observed, based on how my own life has changed in the last few years.
Building for the model
We all know that models perform better with more data. The diagram below is an oversimplified, albeit useful, illustration of this virtuous cycle 👇
AI has gotten a LOT better now. This makes the flywheel more important and more broadly applicable than ever, leading to a few added responsibilities for PMs, like
- Understanding how model performance correlates to product performance — taking part in the process of choosing a cost function (what the model optimizes for) and deciding what to measure are both crucial in letting AI help your product. The entire flywheel breaks when the wrong objectives are set.
- Building for the model, not just the user — this means understanding what signals the model needs and ensuring that the product is able to collect them. Apart from driving user value, the product needs to collect data (ethically!) that can be used to make the model better. Think 👍 👎 buttons or any actions that clearly signal whether the user likes what they are being shown.
The only thing worse than building a product that does not work is not having enough data to know if the product is working. For products that use AI, this principle is now truer than ever. The model is like a separate stakeholder. As PMs, we need to think about meeting its requirements.
As an example, think of why Tiktok does a great job in recommending the next video to you. One of the main reasons is the product’s UX, which makes the user watch or dismiss every recommended video. Tiktok’s interface, where the user is forced to be opinionated, provides richer signals to the model, than a feed that shows multiple videos at once. I love this article by Eugene Wei that talks about this idea in more detail.
Getting involved in the ML pipeline
I had to upgrade my cursory knowledge of ML to really understand how these techniques were being applied. Then, I started getting involved in areas that I would’ve usually labelled as “engineering team territory”. This includes reviewing the cost function, deciding how the model will be evaluated, helping out with prompt engineering, defining how to default when the model doesn’t work and defining “ideal” test data for the model.
Each of these topics warrants an article of its own. Take model evaluation as an example — defining how the model’s output will be evaluated is one of the first steps in building the pipeline. If you’re working on a customer service chatbot, you might want the model to be helpful, respectful and understanding of the user’s needs. Then, “helpfulness” and “respectfulness” become factors that you evaluate the model on. Human raters will be given a set of N responses from the model and asked how helpful and respectful each response was. Before using this model with real users, you might want it to be helpful at least 90% of the time and respectful 99.9% of the time. This is an example of how it’s part of the PM’s job to think beyond success metrics and think about what metrics are appropriate for human rating.
̶S̶a̶y̶i̶n̶g̶ ̶”̶A̶I̶ ̶i̶s̶ ̶a̶ ̶t̶r̶a̶n̶s̶c̶e̶n̶d̶e̶n̶t̶a̶l̶ ̶s̶h̶i̶f̶t̶”̶ ̶i̶n̶ ̶f̶r̶o̶n̶t̶ ̶o̶f̶ ̶t̶h̶e̶ ̶m̶i̶r̶r̶o̶r̶ ̶e̶v̶e̶r̶y̶ ̶m̶o̶r̶n̶i̶n̶g̶
Being opinionated about when/how AI can be useful to your product
There is undoubtedly a solution-first mindset being adopted across the industry right now. Several organizations and teams have chosen the AI hammer, which makes every problem look like a nail. AI poses serious ROI questions for a lot of use cases due to both unreliable returns (quality gaps, hallucination) and large investment (compute costs). However, it’s clearly a once in a lifetime technology that can provide deep meaningful value when deployed for the right purposes. As PMs, our responsibilities extend to evangelizing the usage and non-usage of AI. Asking uncomfortable questions like “Will using AI truly lead to increased user value and are the costs worth it?” are some of the biggest favors a PM can do to the team right now.
The exciting/scary part about this space is the speed at which it is evolving and how different job families are trying to keep up. I’m not sure how this article would’ve aged by the end of 2024, but I’m very happy to find out the hard way.
Disclaimer: The opinions stated here are my own, not necessarily those of my employer.
Please consider subscribing to my substack if you liked this article. Thank you!
