De-risking innovative AI products
Part three of the Prosus AI series on how to achieve AI-by-design product and business innovation
This blog is the third part of a series; find parts one and two in the text below and on our Prosus AI tech blog.
Liesbeth Dingemans, Serena Westra
At Prosus AI, we have been working for almost two years on developing an approach to 'AI-by-design innovation'. Our dedicated Squared team has focused on answering one central question:
How to spend a part of your data science capacity on discovering, testing and launching riskier, longer-term AI applications — with a potential to disrupt your product and create 10X business impact?
Previously, we've written that it takes a carefully organised, dedicated team to translate a high-level goal as 'AI-by-design innovation' into clear and testable product proposals. We've also highlighted that finding innovative AI concepts is centred around a deep understanding of your customers' needs — and the potential for AI to radically solve their pain points.
What distinguishes a good innovation approach from any ad-hoc brainstorm meeting, is the ability to quickly translate proposals into small and testable chunks. In this blog, we cover the fourth step of our Squared Initiative approach: test. This is where it’s time to move from being creative to being practical.
Building AI-by-design products has many similarities with normal product development, but also requires solving a number of unique challenges such as:
- How do you quickly test a radically different customer experience, without investing months of development time?
- How do you check the feasibility of an AI product proposal for which you might not even have the data, let alone an optimally performing AI model to drive the experience that your team has designed?
- How much time do you spend on researching and de-risking your product idea or running various kinds of experiments — and when do you decide to simply take your chances and build the full end-to-end product?
These and more questions will be addressed in this third blog in the Squared series on AI innovation.
What distinguishes a good innovation approach from any ad-hoc brainstorm meeting, is the ability to quickly translate proposals into small and testable chunks
Getting started with testing: defining your riskiest assumptions
After your team has gone through rounds of concepting to get to a winning AI product proposal, you probably have a feeling of victory! The team has identified at least one promising concept, and everyone is eager to get their hands dirty. It is a real challenge not to lose yourself in the concept, and maintain an open mind. But diving in too deeply too fast increases your risks of crashing into an avoidable wall down the line. Just like any product proposal, an AI-by-design product proposal has different types of risks:
- Viability risks (is it a smart move for our company?)
- Feasibility risks (can we technically deliver this?)
- Desirability risks (do customers want this?)
Quickly understanding these risks, mitigating them where possible and accepting it when risks are too large, will help the team to keep moving fast and avoid getting stuck in dead-end ideas.
The more time and money you spend on a concept, the harder it gets to “kill your darlings’’. This is also known as the sunk cost fallacy. Human brains are wired to avoid losses, which can result in an ongoing commitment. It might even happen that the team starts blaming their customers instead of themselves: “these customers just don’t get it.” To avoid this, it is essential to have clear success criteria early on in your testing journey. Breaking down your product proposal into assumptions that can be refuted and KPIs that can be measured, helps to keep the team honest about the real potential of the concept you are testing.
Quickly understanding the risks of a proposal will help the team to keep moving fast and avoid getting stuck in dead-end ideas
With our Squared team, we identified the riskiest assumptions of the concept we were working on: which assumptions need to be true, in order for our concept to succeed? These assumptions can be related to the desirability, feasibility, or viability of your concept. It is a common mistake to start with the low hanging fruit: to make impact in a short amount of time to show progress. Rather, you should start with the important assumptions, for which you have least evidence. If these challenges are solvable, the chance of a successful concept is high.
Imagine you want to research if there is potential for video in buying and selling second-hand clothes: TikTok meets classifieds. It is tempting to spend your team's time on learning what the most attractive video looks like for selling a second-hand dress and start testing features for video editing. But the assumption "We believe that we should offer video editing tools to help sellers create appealing videos" is not the riskiest one. A much riskier assumption would be: “We believe that sellers are more successful when adding a video of an item to their listing”. This assumption touches a core KPI — namely a seller's ability to make a successful sale.
Where to start: desirability, feasibility or viability?
Developing innovative AI-by-design products is in some ways similar to normal product development: for example, desirability of the product is still essential. When customers are not interested in your solution, it doesn’t make sense to invest development time and your product will likely not generate revenue. Therefore, we usually define and de-risk our desirability assumptions first — do customers even want this? Does it solve some of their core pain points or play into their deepest needs — with a potential for 10x impact?
What’s different when aiming to create radical innovation, is that you're dealing with hypothetical customers of a not-yet-existing product and their future needs. The fastest route to de-risking is to break down your solution into smaller assumptions, which are testable with existing customers of your product. Most often, you can fake it until you make it. You don’t need to build an actual tool to find proof: with a prototype you can already measure if customers are interested. This does require a new mindset: willingness to expose some of your current customers to a radically different product direction.
There are many possible tests to de-risk the desirability of your product direction. Think about one-hour in-depth interviews about a prototype, surveys, fake door experiments, Wizard of Oz or A/B tests. One test will not give you all the answers and certainty you are looking for. Therefore, it is important to test multiple assumptions in parallel. Start small. The more certain you are about your solution, the higher the fidelity and the larger the scale of your tests can be — creating more reliable and quantified evidence.
We learned this the hard way. We created a high-fidelity prototype: a clickable version of the app, which simulated the new customer experience as real as possible. The team put time and careful consideration into it, and even designed a unique look and feel… only to find out that customers didn’t really like the idea behind it. In hindsight, it would have made much more sense to first test the general direction of the solution, instead of wasting time pursuing an idea that was not as desired as expected.
To create radical innovation, you deal with hypothetical customers of a non-yet-existing product and their future needs
How to test AI solutions?
Even though we recommend you to research and test the desirability of your product direction first before spending too much time on building AI models, your proposal also needs to be feasible. We have learnt that AI-by-design innovation doesn’t have to be driven by radically innovative AI. What is typically innovative, is the interplay between product and AI. The best ideas allow you to optimise your product to collect laser-sharp signals about your users, that accelerate your learning about their needs and continue to improve your AI (also known as data-centric AI as introduced by Andrew Ng).
Even when your AI model itself is in theory not radical, it can still be a challenge to feel certain that your proposal is feasible — especially if you don’t even have the data to train your model. Here are a few lessons that we learned:
- Be creative with small data samples. We have often combined open-source models with a few data samples of our own to approximate the future start of our AI model.
- Design data collection solutions first. When the data that you need in order to drive your product experience doesn’t exist yet, the first version of that product can be aimed at data collection instead of automation. AI can be added later, to achieve your full product vision. For initial tests, it is often possible to fake the full product experience using heuristics. (For example, check out this old blog by Google — where they image what it could be like if 'you knew the exact traffic speed on every road in the city'. They first collected driving patterns, and later used this to optimize their route suggestions and traffic predictions.)
- Build demos that incorporate both the product and the AI components. In our Squared Initiative, we often build data apps using Streamlit to collect early feedback on our models and their user interaction. We learned that creating an AI demo does not only help to understand the technical feasibility of the concept, it also brings the concept to life. During our stakeholder meetings, the AI demos helped to leave a lasting impression and show the potential of our innovation ideas.
These feasibility checks are helpful when you are working with innovative AI concepts. More standard feasibility checks should also be considered, such as researching whether other teams have ever tried to achieve something similar (if not, your idea might be less feasibility). We also recommend discussing how often the AI model would need to be correct in order to create the desired experience. If the cost of even a single wrong prediction is high, the feasibility will be lower.
For initial tests, it is often possible to fake the full AI-by-design product experience using heuristics
What about viability?
When you build products, your goal is clear: bring your vision to life and launch your product. This though, is a challenge for our Squared team. Our team is not designed to own, monitor and optimise products in the long term. Our goal is to discover and test AI-by-design innovation in our businesses but leave it to existing or new-to-be hired teams to scale and run these products. This means that our goal is typically to convince stakeholders to invest into one of our concepts, by dedicating a team to scale it. This makes the end point of our team’s process a lot less clear: when have you collected enough evidence to be sure that your AI-by-design innovation will work?
How much research and testing it will take to convince your stakeholders depends on factors such as your company’s willingness to take risks. AI demos and Wizard of Oz tests help to create a positive buzz for an AI-by-design concept. But our Squared team soon discovered that to convince C-level stakeholders, a prototype test with 25 users is not enough. We had to adjust our way of working to be able to measure and quantify the potential and demonstrate product-market fit. We integrated experiment design into our core capabilities. We hired an engineering manager and developers. We now aim to push our solution from prototype tests to real, live experimentation and building end-to-end MVPs.
What comes after the 'test' phase?
When we launched the Squared team, we thought that our missions would end with an 'inspire' phase: inspiring teams with what the AI-by-design future of your business or product could look like. We came to realise that “inspire” is not the final stage of the innovation process: it should happen all along the process.
Keeping the rest of the organisation involved in your innovation can be challenging. How do you know at the start of the project, when the scope is undecided, whom to involve? Involve too many stakeholders, and you will slow your innovation down due to a high amount of alignment meetings — and high risk of ideas getting killed based on 'gut feeling' rather than real experimental evidence. Involve too few stakeholders, and you will have difficulties to find a sponsor to adopt your concept and build it. These and other challenges will be discussed in future blogs.
Stay tuned for the next edition and feel free to get in touch with us at datascience@prosus.com to continue the conversation on what AI innovation really means, and how to achieve it.
We’d like to thank our colleagues from the Prosus AI team, OLX Group and Koos Service Design for their suggestions and help in editing.
