The role of a technical program manager in Generative AI products
With contributions and reviews by Dan Massey and Mona Soliman Habib
Not too long ago, I wrote about the “The Role of a TPM in AI Products” and “TPM considerations for Machine Learning projects.” The world changed on November 30, 2022, with the launch of ChatGPT. Words like AI (Artificial Intelligence), LLM (Large Language Model), and GPT-4 (Generative Pre-Trained Transformer 4) became household names. Developers, designers, and PMs started integrating AI tools like GitHub Copilot, Figma, DALL-E 3, and Microsoft 365 Copilot into their planning, inner, and outer loop workflows. AI was no longer a research playground — it was open for business!
Since 2022, we are seeing applications of AI in every industry; many of them are transformational and not just incremental or sidecar features.
With these rapid developments, the future definition of a product team might also change. Headlines like “Airbnb ‘eliminated’ the traditional PM role. Now what?” reflect how companies are already seeing a shift in roles and responsibilities. So, what does this mean for the technical program manager (TPM) role?
In this post, I attempt to answer the following questions:
- “Where should I focus as a TPM now?” (a.k.a. Do I have a job anymore?!)
- “What are the considerations for a TPM in building Generative AI products?”
- “What skills do I need to be an effective TPM for Generative AI products?”
TPM versus Generative AI TPM
Let’s address the obvious problem: Is there a Generative AI TPM role required for building products?
Let’s answer this question from an AI product perspective: Has the development lifecycle for products changed? Here is a view of a typical AI Lifecycle as shown in the Generative AI — Microsoft Solutions Playbook. The typical stages listed such as Data Preparation & Curation, Experimentation & Evaluation, Validation & Deployment, and Inference & Feedback loop remain the same.
Yes, the introduction of Generative AI platforms like Azure Open AI and powerful models like GPT-4 are intended to benefit the development of AI-enabled products and help us realize opportunities that may be costly if undertaken via traditional ML and other methods. However, the fundamentals of “Why are we building it?” and “What should we build?” hold the same importance as before.
Generative AI will not define your business problem but will assist in building the case and implementing it.
To summarize, the core TPM role does not vanish with Generative AI. A TPM is crucial in landing the right use cases, aligning the teams, and then identifying how solutions may include Generative AI in ways that are cost efficient and responsible.
A TPM, however, must also be aware of how to benefit from the Generative AI opportunity, identify where it creates value and where it diminishes the returns. TPMs must upskill themselves to understand considerations when dealing with use cases that may leverage Generative AI. The remainder of this article focuses on those specifics.
Aligning the Generative AI opportunity with business
As a hypothetical illustration, let’s say that after a “Generative AI 101” seminar, Dr. Max Wellton, CEO of Large Health Company (LHC), pushes the company toward AI, demanding a Health AI Copilot within a month. At first, developers are excited to code and quickly learn Python, Semantic Kernel, and PromptFlow to build a rushed Copilot for healthcare professionals. But the excitement soon turns to frustration for end users because the AI Copilot requires at least four prompts to complete any task and cannot be fact-checked for patient data due to compliance issues, leading to low adoption and ultimately the solution being scrapped. Dr. Welton’s is now on his next adventure, “Oops, AI Did It Again: The Generative Misadventure.”
This story reflects the growing pressure on product teams from executives and an equal urge from developers to use Gen AI and do so quickly. Yes, AI has a lot of potential, but without a proper use case and understanding of end user pain points it can quickly turn into a technical debacle.
Aligning everyone to solve a business problem so that we don’t create a “for-the-sake-of-AI” product is the primary job of a TPM.
Balance the hype: As with any other product, TPMs must weigh Generative AI’s potential against realistic business outcomes. TPMs must collaborate with UX researchers and designers to understand customer needs and validate business requirements. Additionally, TPMs must gather market data to justify AI investments using customer analytics from feasibility studies and present cost-benefit analysis to leadership to make informed decisions.
Understand Organization maturity: As much as your leadership may be excited about building Generative AI products, you must determine the maturity of your team, business unit, and organization before you commit to the product.
Things to consider:
- Develop organization confidence measures: These define metrics such as clarity of business goals, agility to deploy to production, security and compliance constraints, readiness of infrastructure (cloud or on-premises), team skills and availability, existing system, and technology state. This approach should provide you with data on taking up a niche technology like Generative AI and a confidence level for success.
- Define value measures: Marty Cagan has a great post on the key risks for building products. The “value” risk is the most important consideration when it comes to Generative AI products. Minimize costs early and leverage tools like user interviews and low fidelity user journeys to validate solutions with end users before creating a product plan or roadmap.
- Say “No”: Don’t feel shy about challenging the need for Generative AI. Many customer problems can be solved with other techniques such as traditional Machine Learning or just automating processes. Avoid creating a technical science experiment.
Start small and measure your growth
A financial services executive in a “FinanceGPT” event announces their firm will use Generative AI in all their banking products in the next year. Then, when the teams start to analyze the work, they realize that the data sources are disjointed, they do not actually have much AI and ML experience, their teams currently have a six-month release cycle, and there are eight levels of compliance approvals before an Azure AI Service can be whitelisted in the environment. Eleven months into the project, the executive releases a note to stakeholders that they will be delayed by another nine months!
This hypothetical story reflects the reality of many enterprises today. With the potential demonstrated by Generative AI and with growing pressure from competition, executives want to benefit from AI and LLM innovation fast (and rightly so!). However, it is important to incorporate a systematic approach from the start. The job of the TPM is critical here to help orchestrate product discovery and engineering, especially when we may not have clarity on the business use case.
Here is a lens on how to apply these to use cases that can leverage Generative AI:
- Start with “Why?”: To re-emphasize what I’ve already mentioned, start with a business problem and the appropriate success measures. At this stage, there is no conversation about a Generative AI product, but instead about the identification of user pain points, the value to be added and, importantly, what success looks like. The outcome here includes business goals, hypotheses, and measures of success, along with market data to use for research on the use cases.
- Know your customer Even before your team starts experimenting with its first model, prepare for end user feedback loops and interviews. These will not only validate your hypothesis but also suggest priorities for user pain points. Do this work in collaboration with the design team through user studies, workshops, and interviews. Define measures of success for these interactions based on the hypothesis and record them for business use case validation.
- Wait, don’t code yet: Once you have initial validation from your customers, plan to run Minimal Viable Experiments (MVE) to measure your design ideas’ applicability and understand the potential use of AI. Additionally, conduct traditional data discovery and run EDA — the reasons for always doing these does not go away. These are less costly ways to validate the feasibility and viability of design choices as well as get an understanding of the data without going through full-fledged development. In the meantime, the developer team can also skill up on AI using free accounts such as Microsoft Azure AI.
- Plan ahead: As the development teams kickstarts development, you must manage stakeholder alignment, backlog prioritization, and risks. For example, for teams to conduct prompt evaluation and domain specialization, the security team must approve datasets to be made available in the development environments. Plan for infrastructure readiness, continuous user testing, data governance, security, and compliance alignment and engineering practices to ensure you can release incrementally.
What should we measure?
Defining success with realistic measures provides confidence in the investments being made and helps pace the organization’s adoption of AI. Dan Massey has an excellent categorization of how to define such measures through the lifetime of a product; here is a sample:
Measures of Generative AI Product Success (github.com)
These metrics, combined with technical metrics for the underlying system such as response time and latency, as well as model metrics such as perplexity, coverage, precision, and recall — among others — provide a set of holistic success measures to provide focus for the team.
Continuous measuring across product discovery and engineering is the heartbeat of a successful product.
Things to consider:
- Think about whether a business problem is truly being presented or whether a solution is what is being offered, merely camouflaged in a business problem wrapper. For more information, see Home — The XY Problem.
- Define and start collecting metrics as soon as users engage with your first ideas. Continuous user feedback not only enriches your features but can also enhance the quality of input prompts and, in turn, response quality.
- Even when you see a clear use case for the use of Generative AI, measure your progress until you see substantial positive customer feedback and organization alignment. For example, conduct relevance judgment exercises for your product domain relevance but also think about non-functional requirements such as fairness and inclusion to avoid hampering credibility with your userbase in production.
- As you start building AI products, it is tempting to get derailed from the original business outcomes and wander into fantasy land. For example, your team might come back and say if I have that large data set, my model can perform better; if we have more compute, we can scale the model better. While these might be valid asks, you must align these decisions to business and success measures. At some point adding more data or compute might have diminishing returns or make the release process too complex or result in a multi-fold increase in the cost of investment, so think about these implications before venturing into optimizations.
- Your Generative AI product may not see the same success as ChatGPT — and that’s OK! Start small and continue to measure your progress with the right feedback loops, which enables you to experiment and move to production with confidence.
- Review TPM Considerations of ML projects, because much remains consistent.
Be the T[eamwork]PM
The “T” in TPM is usually identified as “Technical” but it’s a loaded “T” (for the curious, see What does the T in technical PM mean?). In the case of AI products, considering the ambiguity involved, the T must include “Teamwork.” At the very least, a TPM should be teaming on the following:
- Garbage in, garbage out: The TPM works with the team during data discovery to ensure the AI is fed with clean, high-quality data, understanding that input quality directly affects output reliability. The TPM also collaborates to set the appropriate level of control over hallucinations tailored to the product’s needs.
- Grounding AI responses: In partnership with developers and data scientists, the TPM explores techniques like RAG (Retrieval Augmented Generation) to anchor model responses in customer reality. This collaboration aims to enhance the model’s responses and may reduce the costs and time associated with other techniques like model tuning.
- Model selection strategy: The TPM collaborates to define the business problem so that data scientists and developers can employ the right model for the job, whether it’s for summarizing texts, translating languages, classifying data, or generating new content. Additionally, TPMs can help frame the choice and configuration of the models by providing end user scenarios; for example, a data scientist may configure the hallucination and temperature parameters for a financial Copilot differently as compared to a Copilot that does content writing based on the degree of factual responses required.
- Establish the baseline operations for GenAI products: In parallel, the TPM needs to work closely with Engineering teams to setup appropriate practices such as LLMOps to conduct quick experiments and convert them into production solutions.
- Cost management: Collaborative discussions led by the TPM focus on the cost aspects of model development, from the decision to use APIs or open-source software to the costs associated with prompt engineering and fine-tuning models.
- Responsible AI practices: The TPM partners with the team to incorporate Responsible AI principles throughout the product lifecycle, ensuring the final product is ethically sound and aligns with best practices for AI safety and fairness. Additionally, the TPM ensures that regulatory compliance and data privacy requirements are captured to ensure transparency with end users on what data is being captured and how it will be used to train the models (if at all).
- Marketing and sales: Collaborating with marketing and sales, the TPM provides insights to shape AI product narratives, guiding teams to understand and communicate the unique selling points of AI and LLM products. Not having your product marketing or sales team in the loop for your AI Product can significantly delay or even stall the launch. TPMs must ensure that they collaborate with product marketing to keep them aware of developments, guide them on the benefits of the AI product, and make them aware of the risks to ensure the marketing campaign covers them. TPMs can also ensure that any legal liabilities from models or data and components used in the AI product are approved and triaged based on company guidelines.
The learning journey of the TPM continues…
I hope this article provides some context on the critical role TPMs can play in building Generative AI products. It’s not enough to be an agile expert or a domain expert — TPMs must understand the essentials of AI and LLM fundamentals to collaborate with their teams and have meaningful conversations with their stakeholders and customers. For more information, see The learning journey of a TPM.
There is a lot of coaching and education that needs to happen in the Generative AI space; the good news is that we are all learning.
The best way to learn is to experiment with these use cases and get your hands dirty in building AI products. Here are some good free / paid resources to get you started on this journey:
- Generative AI — Microsoft Solutions Playbook
- GitHub — microsoft/AI-For-Beginners: 12 Weeks, 24 Lessons, AI for All!
- Career Essentials in Generative AI by Microsoft and LinkedIn
- Microsoft Azure AI Fundamentals: Generative AI — Training | Microsoft Learn
- Generative AI Fundamentals | Databricks
- Databricks: Large Language Models: Application through Production | edX
- GitHub — microsoft/generative-ai-for-beginners: 12 Lessons, Get Started Building with Generative AI 🔗 https://microsoft.github.io/generative-ai-for-beginners/
- RAG (Retrieval Augmented Generation)
- Responsible AI toolbox
Happy learning!
Nikhil Sachdeva is on LinkedIn.
