4. How to choose the right problem to solve? (Goldilocks problem) / AI Product Management
Incorporating AI into the products in your organisation doesn’t mean utilising machine learning or deep learning algorithms to solve every problem all at once. It’s important to pick a single hairy problem to start with, take learnings from solving that problem and build momentum to extend the AI capabilities across the business.
That’s called identifying the “Goldilocks problem.”
For example, in the previous post, we talked about building an AI solution to cover 60% of the informational queries from the customers, hence reducing the response time on these queries, and improving customer NPS and lifetime value (LTV).
Building a fully automated customer care product would be the ultimate aim for the said business, however taking care of 100% of the customer care queries would mean building for 1000s of use cases to cover all the scenarios, with triaging to a customer service personnel whenever the queries get too complicated.
Hence, the focus on 60% of the informational queries — identifying the problem that can create a significant business impact — is critical to the initial success of AI.
Even if the model only successfully recognized 75 percent of the inbound informational queries related to bulk discounts and international shipping at the beginning, that still contributed meaningfully to the bottom line of reducing customer response time. Over time, the model can be improved and expand its capabilities to identify all distinct use cases.
The most important thing you can do to set yourself up for success is to choose the problem that hits the sweet spot of scale and impact with a manageable machine learning component. So what are the characteristics of this Goldilocks problem?
1. Well-defined and limited scope
The best Goldilocks problem is limited in scope so you can solve it quickly.
A problem that involves easily classifying into a few buckets is a great candidate.
For example: Is it an informational query — is it about bulk discounts or international shipping — yes or no?
These types of problems have solutions that are easy to codify and hence the solution doesn’t typically fall into a grey area (as opposed to taking a judgement call on whether to take an action).
Start simple. Something that a human can do reliably over and over again, but that a computer could do at a scale and speed a human could not, can be a good problem to choose.
2. Data Availability
A good Goldilocks problem in the context of AI has a large chunk of historical data associated with it, and many examples of solving that same problem in the past.
Informational queries in the customer service problem meet this criteria. For example, the business in question will have a large amount of correspondence between customer service personnel and customers enquiring about bulk discounts and international shipping. All these past email correspondences can serve as training (and testing) data for your model. While selecting the data for training the model(s), it is important to ensure that data spans a range of use cases and doesn’t unintentionally introduce bias or unfairness.
3. Quick Wins
Your Goldilocks problem should demonstrate the value of AI for your business in a short timeframe.
With price and speed to deploy in mind, a few factors you can consider are
- off-the-shelf models
You may have a use case that can be solved in part or whole by an off-the-shelf model. An off-the-shelf model is one that someone else has already developed and is selling as a service. This means that the model comes pre-programmed and matches the specific problem you’re trying to solve.
An off-the-shelf model will help you demonstrate value quickly to the business, thus enabling you to make a case for a larger investment into AI across a number of business functions or the whole company.
When trying to pick this first problem, you should consult with the data scientists and other team members to see if there are opportunities to take this kind of shortcut.
2. off-the-shelf training datasets
Using off-the-shelf training datasets can be a quick and cost-effective alternative to collecting and annotating data from scratch, even if you are building your own model.
High-quality datasets can be used as-is or customized for specific project types. These datasets are offered by companies that guarantee accuracy up front, thereby removing the element of variability from the model training process.
Aside from the obvious cost and speed benefits, the growing demands for data privacy and security from both customers and authorities may make it difficult to utilize data you already possess.
Whether you should buy or build data sets and models within the organisation is also a broader organisational AI strategy question, which we will discuss in a future post.
4. Clear, measurable impact
A Goldilocks problem even though limited in scope, should still have a clear business impact.
A good rule of thumb is to not only be clear on the business impact but be able to measure and prove it clearly. Our customer service responses to informational queries perfectly fit the bill. We could correlate the reduction in response time to repeat purchases and an increase in customer lifetime value (LTV).
Often, Goldilocks’ problems are linked to obvious things like revenue, customer net promoter score (NPS), or time value.
It’s also helpful for the solution to be novel or innovative in some way, to get the entire organisation excited. If teams across the organisation get excited about what the AI can do, then they are more likely to get involved by coming up with more problems to solve and supporting the AI team who works on them, thereby fostering a culture conducive for AI to succeed.
The data maturity of your business — the amount of experience with data, analytics and AI solutions and the degree to which it trusts data to help it make decisions — will help you determine the Goldilocks problem you pick.
Some organisations may not have well-defined or clean datasets that help them with decisions, resulting in no trust in data at all. Some employees may also have concerns that deploying AI models can lead to jobs being eliminated and redundancies.
Demonstrating the value of AI with an easy win can go a long way toward helping organizations mature into data-driven companies that trust data to help them make decisions, and as a result, are willing to use AI to tackle larger problems.
More practically, companies that don’t have a lot of experience with AI will need to start with something easy before moving on to harder problems. This can act as a risk mitigation strategy, wherein if it doesn’t go quite right, won’t cause major harm or embarrassment to the business. Again, our information query response model is an excellent example of this concept. Even if the NLP classifier didn’t work at peak accuracy and several informational queries were misclassified, it would certainly be inconvenient but not catastrophic.
In summary:
In the next post, we will discuss how to define the success metrics to track.
See ya next time!
