EXPEDIA GROUP TECHNOLOGY — DATA
AI-First Product Management: Framing
A framework to move your AI-first product from concept to production
If your company is developing its AI strategy and you are the product manager assigned to build one or more of its AI initiatives this post should help you. Do you have good knowledge and experience of the AI space already? Perhaps you worked as product manager on a couple of innovative initiatives, yet an AI-first product looks like an entirely different journey? I’ll lay out here the framework you need to move from concept to production.
A little bit about myself: I am a PM at Expedia Group™ working on a Data Science and Algo team as part of the Marketing Organization. My day to day job is helping my business understand customers, reach out on the right channels and present them with the best travel options. For that we produce billions of predictions per day using AI, run A/B testing and apply statistical rigor in everything we do. I’ve also had the opportunity to work on other types of AI-first products in the past: conversation understanding as part of LivePerson Conversational Cloud and also ranking and recommendations engines for Azure Marketplaces. This has given me the opportunity to be involved in a good variety of AI products combining ML techniques, NLP pre-trained models, transfer learning and more.
In this post I want to give an overview of the work involved and what can make or break your effort: bring the right expertise at the right time, how to drive ideation and most importantly how to create structure and clarity as product manager.
Executing an AI strategy takes a multiple quarter roadmap so getting your team to “Think Big and Small” instead of incubating is what it takes to succeed. Here’s what I’ll cover in this first part of this two-part series:
- Focus on “AI First”
- Customer development
- Business framing
Your journey gets the most exciting when you take your product from offline simulation into A/B testing to create a feedback loop to improve. Here’s what I’ll cover in the second part of this two-part series:
- Ideation and prototyping
- Test and learn
- Simplify to productize
AI-First comes to your rescue
Let’s ask ourselves, what defines an AI-first product? In my mind there are three things that will set it apart:
1. Your business problem reached “the glass ceiling”: rules are unmaintainable, decision trees are not granular enough and human expertise can’t scale anymore.
For example: Can plain old analytics and forecasting meet business needs? How far off are the results with such a simplistic approach? Are you in the position of changing just one rule and at that point no human can tell if it still works? If yes, you are looking at an AI-first urgent step-in.
2. Your data is large and captured in great granularity. It deeply captures your business processes with individual interactions from start to end.
For example: Let’s imagine you have to create a recommendation engine. You’ll have to answer with yes to many questions before starting any work: What are all kinds of data you would need and are they available? Do they go far back enough? Can you use this data right away or does it need a few months to build a consumption pipeline? Do you have ethics practices in place for that data? If you answered yes to these questions, you’re ready to start.
3. Your AI-first product integrates into a larger AI strategy. You have data engineering, a machine learning team and domain experts that pertain to your business problem: linguistics for NLP, business analysts for marketing etc.
For example: If you work in a large organization like Expedia Group you may have it all in place and you can get your roadmap and collaboration started. If you are working for a startup and can only have a tiny team, think how you will bring expertise: consulting, partnerships, etc.
Now you have got your three pillars in place: the challenge, the data and the talent. Time to think about your customers. We always start with them!
Talk to customers to learn what data matters
The first step goes without saying: know your customers. Is this a directly consumer facing solution or is it reaching them via a B2B process? Either way, you may learn your solution can power multiple products. Think model as a service when pinning down requirements.
Here’s an example of a directly consumer facing product: your recommendation engine shows personalized results on the homepage of your marketplace. You should not stop here. Can your Salesforce solution use the very same engine to find and prioritize proactive reachout? Expose the confidence score of your predictions, A/B test the optimal threshold and… voilà you are now powering two products.
For B2B products, your customer engagement can get even closer. Let’s imagine they are customer representatives in a care center. They have to audit conversations to improve or automate how are they are addressing the customer problem. Now just ask yourself, how do they get this done? There are two steps to find the answer:
Step 1: Shadow your customer for few hours. How do they analyze the data? Do they use regular expressions, excel spreadsheets, other tools?
Step 2: Try to do the job yourself and see what it takes. Can you even do this work and maintain quality for more than few hours? As a bonus, you’ll learn about what types of data matters along the way.
You’ve done all this awesome work and engaged with your customers. Now you’re ready to start framing your problem:
Yes, your model can boil the ocean… do that later
We tend to think our AI product needs to solve all the customer problems at once like a magic wand. On the other hand, connecting to your customers has shown you can start with just one type of problem and do a good job. For example, you can recognize one type of conversation, image, etc. and deliver immediate value. If “intent to unsubscribe” is golden to your customer, start with that. This exercise will show your team where the bar is for winning against your customer’s substitute/workaround. You may be surprised how easy it is to have a quick win! Most likely you don’t need 99% accuracy on day one to double the efficiency of their business. On this point, I can’t stress enough the importance of bringing specific business requirements from your customer. What one type of prediction does your product needs right now? And what is an acceptably accurate prediction? To break this down further here’s a way for you to define the type of problem you’re solving and metrics that will define your success:
Does your product need to understand existing data?
Example: NLP problems utilize a taxonomy/ontology to map natural language to establish concepts: intents, entities, etc. Image recognition tries similarly to classify. For all data that can’t be classified it will try to auto-cluster and help customers navigate it easier. If those clusters are unmanageably large, one way to explore them is by using a dimensionality reduction technique and visualize.
If data was not properly classified you’ll have to explore the gap, create required data and reinforce your model. Sounds simple on paper yet finding the right data and eliminating bias in labeling will need rigor and great statistics skills on top of the domain experts that define your taxonomy. Engage all this expertise in the team early on!
Does your product need to create new data, or any digital content? (text, images, movies, etc)
Example: In image processing field, it would mean to increase resolution into an image or fill in areas where light was too strong or not enough. For more complex problems like an industrial process or serving as an actor in an auction, we need an objective function/formulation, and the values generated may be on a continuous scale rather than discrete. In these problems, you only know your past product’s actions as an actor. You don’t know what all the other actors did and further will do or even what is their objective. This is why you can’t even go into the past data and simulate a full what-if scenario. There is also no human/auto-labeling or human crowdsourcing to leverage directly.
Let me give you another example: you want to personalize the experience of a user and introduce recommendations. You’ll likely formulate this as an optimization problem yet you can never walk back in time and try again until the user engages. You’ll only know the error of your prediction. This becomes an interesting problem of what data signals you should bring in to know more about that context. This is where hypothesis based experimentation will get you moving!
At this point you may feel your AI-first strategy is just about to change the world. In my experience you should not try to “boil the ocean” yet. I’ll talk in Part 2 how to gradually increase scope and performance for your Ai-First product. For now I’ll close with three questions for you to think about:
- How much should your customers know about AI?
- When should you bring your science team into customer sessions?
- What should your science team know about the business case before starting to work on the solution?
