Does Your Virtual Assistant Actually Cover What Users Want?

Eric Wayne
Mar 4 · 4 min read

The newly enhanced Watson Assistant Intent Recommendations helps you to quickly expand your coverage

Image for post
Image for post
Photo by Omar Flores on Unsplash

Have you checked your coverage lately?

Creating a virtual assistant is easy. But providing one that customers or employees actually want to use and that provides value to your business takes time and effort. Users interfacing with these AI systems expect instant gratification — they have a job to get done and want to get to a resolution right away. An assistant that keeps answering “Sorry, I’m not trained on that”, will frustrate your customers and chances are they will walk away and not come back.

We find that successful projects include an ongoing focus on both the quality and coverage of the assistant. You can think of quality as how well the assistant performs the specific job it is designed for — to answer questions, search knowledge bases or trigger transactions. Coverage is the portion of questions or actions your users request that you have already trained your assistant to handle. Low coverage means that your users will frequently see a response like “Sorry, I’m not trained on that.”

The Watson Assistant team just released a new Intent Recommendations feature that uses your live assistant logs as the source for new intents and user examples. The feature helps you increase your coverage by adding new intents and user examples based on what your users are actually asking about in production.

An Example
Imagine you are a telco provider and your assistant helps your customers pay their bill. After you’ve scaled out your assistant, you are finding that many interactions are still being transferred to a live agent. Examining your assistant chat logs, you find that customers are frequently asking about charging their phone or requesting to change their address — tasks that you didn’t anticipate. Adding these intents to your assistant’s scope will increase your coverage and lead to an increase in conversations that your assistant can handle on its own.

Examining your production chat logs to find gaps in your intent coverage sounds simple, right? The challenges arise as a price of your success. As you scale your assistant to more users, you inevitably begin to drown in the volume of logs to sift through and assess. You need to identify how different user utterances form related patterns. To prioritize which ones to add, you need to understand which are being asked about most frequently. And you need to separate noise from training data that would actually benefit your coverage metric.

What if AI could help automate this job? These challenges are what Watson Assistant’s new recommendations feature is designed to address. It reduces the time and labor required to improve your coverage. All this in 4 steps.

How does it work?

Step 1 — Choose your source: You can get started today by simply opening the dialog skill you wish to improve, clicking on the Recommendation Sources button, and connecting your live production assistant as the data source for recommendations.

Recommendation source selection

Step 2 — Explore recommendations: Watson Assistant automatically takes care of sifting through your chat logs and grouping utterances into candidate intents. The Recommendations feature automatically filters out noisy user utterances and uses unsupervised machine learning techniques to group candidate intents together. Then a ranking algorithm places the intents most likely to address gaps in your coverage at the top of the list. You can see the number of log utterances for each recommended intent in the list. This number is an indication of how frequently these types of utterances are found in your user conversations.

Image for post
Image for post
New intent recommendations

Step 3 — Add new intent: You can choose whether to create a new intent or add selected utterances to an existing intent. As you add to your training, the recommendations are automatically updated to highlight any remaining gaps in your training.

Image for post
Image for post
Add candidate intent with selected examples as new or to existing intent

Step 4 — Expand coverage of an existing intent: If you’ve determined you need to expand the scope of an intent that you already have, you can do so. Open the intent and click Show recommendations to see a list of intent user example suggestions. The recommendations will display utterances from your assistant logs that are similar to other user examples in that intent. This feature employs an algorithm that follows a principle of “more like this,” helping you find additional ways of expressing the intents you already have. Select user examples and click add to expand this intent’s training data.

Image for post
Image for post
User example recommendations

Get started today!

Build an assistant that your customers will love and let AI do the hard work for you. Use Watson Assistant’s new intent recommendation feature to expand the breadth of topics that your assistant can help people with. To learn more about improving the coverage of your AI assistant, explore the following resources:

· IBM Watson Assistant Continuous Improvement Best Practices guide which provides a roadmap for measuring, analyzing and improving your assistant’s coverage and quality

· Get help defining intents in the IBM Watson Assistant Product documentation

IBM Watson

AI Platform for the Enterprise

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store