How to Drive the Right Outcomes with AI for Your Products

AI practitioners are all too familiar with statistics that over 80% of AI projects fail. A lot has been said about what organizations and data science teams can do to increase this low success rate. Nonetheless, even organizations with established machine learning (ML) practices and high-end AI teams struggle. Some AI initiatives become transformational for the business, while others show little return on investment or never even come to fruition.

Of course, this isn’t unique to AI projects, but since data science is a fairly new discipline, there’s another factor impeding success: not everything can be solved with AI. In this post, I’ll explore the problems we can expect AI to solve well, drawing from my firsthand experiences as a data science group manager for Intuit QuickBooks.

The first thing to consider is the outcome we are trying to achieve with the machine learning system. What are the expected benefits of the model to the end user experience? What metrics are we using to measure success?

For example, the entire system may be expected to affect a business metric, such as the time it takes a user to complete a task. But, as data scientists we usually look at a statistical measure when we evaluate the model itself. For example, AUC (area under the receiver operating characteristic curve) or precision. Bottom line? The entire system should be designed so that improving the model metric yields an improvement in the business metric.

From personal experience, most models typically end up serving a small list of objectives, and for each objective, there are certain considerations to address to set the project up for success. So, let’s dive into the most common ones, keeping in mind that this is a snapshot for AI practitioners and is not intended to represent every possible use case.

Boosting productivity via automation & augmentation

Enhancing productivity is generally the use case with the best fit for AI. This includes all scenarios where there is an existing process that is performed manually and the system’s goal is to make it more efficient. The benefit here is very clear and can be easily measured by tracking the human hours saved by the system. Classic examples are auto-suggest or search systems, but even a smart algorithm that routes customer calls to support agents can increase productivity in the context of the entire system.

Of course, we need to ensure the process is worth automating. To answer that question, we need to consider the following:

• How many users are performing tasks as part of the process?
• How much time do those users invest in the process?
• How much of that time can the AI system save?

If the scale of the process is large and AI can reduce a major part of it, then it is probably a model worth building. Nevertheless, there are two additional factors to be considered when designing an AI system to increase productivity:

• First, is the AI component fully automating the process, or is it increasing the throughput of the people involved in the process while keeping them in charge?
• Second, is it an internal, back-office process, or a customer-facing experience?

As machine learning is by nature a statistical method that will inevitably make mistakes, the system needs to be designed with fail-safe mechanisms. It is much easier to design such mechanisms for back-office users, as you can educate them about the model so they understand the overall productivity benefits. On the other hand, when an AI system is automating tasks to make end users more productive, we should design it in such a way that instills end user confidence in the output of the AI.

To illustrate this, let’s consider two examples of productivity-enhancing AI models we have built at Intuit.

• When a user imports transactions from their bank account into QuickBooks, they need to categorize those transactions into the correct account in their books. To save the user the effort of manual categorization, we’ve built an AI model that suggests the right account for each transaction. As this is a customer-facing process, and we want the customer to feel completely confident in the AI, the system is designed so that the user must approve the suggestions, thereby remaining in control throughout the process.
• On the back-office side, we have created a natural language processing model that is able to summarize calls for our customer success center. This model reduces the need for customer success agents to take notes during the call and enables them to give their full attention to the customer. The AI is fully automatic, and the summary is generated without any input from the user, which boosts the productivity of the process. This is possible since the recording of the call serves as a built-in fail-safe mechanism and the customer success agents are trained to use it in case the output of the AI is not sufficient (e.g., not clear enough, detailed enough, or wrong).

Enhancing customer experiences with personalized recommendations & insights

This set of use cases includes models that proactively surface pieces of relevant information to the user in order to unlock a completely new (and improved!) customer experience.

Recommendation engines are the most well known technology of this type, suggesting your next binge-worthy Netflix series or favorite new product on Amazon based on your viewing and purchase history. These systems provide us with useful information that makes our experience in the product more delightful. Insights, however, can be even more general. For example, an insight can suggest how to price your item you’d like to sell on an ecommerce site or recommend the best time to leave for work.

At Intuit, we provide small businesses with insights and recommendations for various decisions in our QuickBooks product. For example, we might suggest which terms are best for a particular invoice or how much cash a business owner should save so as not to run into liquidity issues.

In each of these examples, the underlying ML model and associated metric can be different for each product, but what we actually care about is user engagement and recommendation acceptance rate. In most cases, a better model leads to a more relevant recommendation, which, in turn, increases the chances of acceptance.

In these types of systems, the intelligence stems primarily from the data. It’s about surfacing the right information at the right time, and it’s the personalization that makes AI the right solution to the problem. So, for any AI practitioner, it’s important to determine early in the process of developing a new product feature whether a sophisticated model will add real value.

3. Improving the quality and accuracy of customer experiences

The goal here is to improve the accuracy or quality of a particular decision-making or task-completion process. If we think of a spell checker, the value it provides is not that we write documents faster but that we write them better. Similarly, if we use a machine learning model to predict the lifetime value of a customer, we are not going after speed. Rather, we’re going after the model’s ability to detect complex patterns in the data and make the computation more accurate, thereby enhancing the overall customer experience

Organizations are constantly making predictions to support various functions. For example, companies predict the price to pay for an ad, whether a customer is going to churn or if a certain activity is fraudulent. In many cases these estimations use many different variables and require extensive domain knowledge to design and achieve high accuracy. These are the cases where an AI solution that leverages existing data can be more accurate than a human.

Error detection products, for instance, are incredibly successful at improving accuracy. These include AI models that monitor the outputs of a process and generate alerts to flag potential inaccuracies. For example, Intuit’s TurboTax uses a deep learning model to validate ACH (automated clearing house for U.S. electronic payments and money transfers) account numbers that users enter for their tax return. Learning from past data, the algorithm can detect when an ACH account number looks invalid and alert the user they might have mistyped it.

In some cases, the AI model can go beyond alerting the customer of an inaccuracy and also suggest corrections. Corrections are produced by predicting the expected behavior, and if our predictor is very accurate, we might consider just inferring the input for the user. At this point, the product development team must carefully weigh the trade-offs for using the model for productivity or for quality, as this decision will have a major impact on the design of the overall system.

Applying AI to solve the right problems in the right ways

AI is a very powerful tool that is transforming industries and creating unrivaled user experiences. However, AI is not a silver bullet to solve all business and customer problems.

AI leaders and practitioners must ensure that AI is applied to the right use cases in order to produce the right outcomes. This is just as important as selecting the right models and extracting the best features.

In Intuit’s case, AI and related technologies are foundational to our mission to power prosperity around the world for consumer, small business and self-employed customers. We identify patterns in our customers’ data to help them find new insights that put more money in their pockets or more time in their day. And, we apply data-driven automation to remove the drudgery out of common financial tasks. By embedding AI throughout our product portfolio — to solve the right problems in the right ways — we’re helping our customers to make sound decisions that improve their financial lives. And, that’s something we’re incredibly proud of as an Intuit AI+Data team.