Product Payoffs in Machine Learning

Sign up for our newsletter here

Uber’s cars are crashing, Microsoft’s bots abusing people on twitter and US judges sentencing people using biased algorithms. We’re using machine learning in important situations but machine learning doesn’t always work.

Machine learning doesn’t always work because it’s inherently probabilistic. That is, machine learning models rely on probabilistic assumptions because they’re trying to model things that are uncertain. Probabilistic assumptions don’t always hold, so the models don’t always work. We should keep this in mind when building machine learning products so that we meet the expectations of our customers, at best, and avoid the unchecked use of machine learning in high-stakes situations, at worst. We may elaborate on the latter topic in a future article.

Payoffs

One way to think about the value of a product is to model the shape of the payoff from using that product. These payoffs are usually in the shape of a curve — convex or concave.

Convex payoffs represent an increase in value as the product is used. That is, the more you use it the more likely you are to have a net positive gain (after incurring the initial cost of adopting the product).

Concave payoffs represent an decrease in value as the product is used. That is, the more you use it the less likely you are to have a net positive gain.

This seems a little abstract but is a remarkably useful demarcator when thinking about where to apply machine learning. This distinction is similar to that between products that earn revenue vs save costs. The difference is that this model focuses on the shape of the payoff, not just that there is a payoff. This brings into focus the magnitude of payoffs at certain times.

Another way to think of this is in terms of interaction frequency. What is the average user experience of a product? If it’s a one-off interaction, for example with diagnosing a disease, then the denominator of the ‘average’ in ‘average user experience’ is 1 so the interaction has to be perfect. The problem is that machine learning is based on probabilities so machine learning systems are almost never 100% accurate.

Examples

Convex

Marketing: AI-based marketing products deliver leads that you didn’t have. Even if these leads go nowhere, there may be low cost and high upside from chasing them down. For example, Clearbit can identify ‘anonymous’ visitors to your website, produce enriched profiles to use as marketing leads and contact details for prospecting those leads.

Sales: AI-based sales products prioritize leads that you may have skipped over. You were going to contact a lead anyway, these tools just increase the chance that the lead you contact will close. For example, InsideSales fed over 100B data points into its Neuralytics engine and increases close rates for customers by 30%. It may not get it right all the time but, on average, you’re more likely to close a deal than not and, in every case, you’re not losing anything by using InsideSales.

Agriculture: AI-based agriculture products can identify opportunities to increase yield by, for example identifying plant diseases and optimizing water use. Using these products will only mean that you grow more using less resources, thus grow more profitably. For example, Acuity Agriculture installs cheap sensors to monitor soil conditions including moisture, salinity, and temperature. Analyzing this data allows farmers to optimize irrigation schedules to save water, reduce runoff and improve plant health.

Inventory: AI-assisted inventory management can identify gaps in stock levels so that you can restock, and therefore sell more. These systems help you fill a hole that was already empty — it couldn’t have got worse! For example, Focal Systems runs computer vision algorithms over videos collected in grocery stores to identify out of stock items so that the staff can quickly restock, meaning customers can buy more.

Concave

Scheduling: AI-based personal assistants arguably save time. However, the ROI calculus is a little lopsided (convex) for customers. The gains from using such a product are dampened because people find it hard to objectively value their time. The losses, however, from using such a product are clearly costly to a customer. For example, if a scheduling bot schedules a bunch of meetings for you, it ‘did the job’. However, if that same bot double-books you and you miss a meeting with an important person then you blame it for the disastrous consequences of damaging your relationship with that person.

Customer service: AI-based customer service bots arguably get through support tickets at a fast pace. However, the ROI calculus is a little lopsided for customers. The gains from using such a product are dampened because the bot will answer a ticket in the most efficient way — there will be no surprise, delight and relationship-building with your customers. The losses, however, from using such a product are clearly costly to a customer because it will elevate the support ticket to a human, who will then have to make up for the frustrating, initial customer experience.

Conclusion

Founders, product managers and designers must carefully think through whether a system with probabilistic assumptions will deliver what their customers need, or if a deterministic system is a better choice. We’re certainly not saying that AI will only ever be useful in certain domains, just that the state of the art means it’s certainly useful in some — not all — domains today. This will all change when we have computers at the edge that are able to perceive and react to the real, uncertain world in realtime, just like us humans can today.

Sign up for our newsletter to be alerted when new pieces are live.