Watson Assistant’s Analyze page: putting data to work
A successful virtual assistant is always improving. Watson Assistant’s built-in analytics are a powerful tool that takes the guesswork out of refining your assistant’s conversations.
Understandable, useful, accurate
IBM redesigned Watson Assistant’s user interface with the goal of getting users from build to deployment in a way that was intuitive, frictionless, and fast. With Actions, its new content authoring interface focused on conversations, Watson Assistant allows non-technical developers to build a virtual assistant in a timeframe that would have seemed unthinkable until recently.
But publication is just the start; a healthy lifecycle for a virtual assistant calls for continuous improvement. Just as the build interface evolved to make creating an assistant easier, Analyze, Watson Assistant’s built-in analytics feature, has evolved to present users with data that’s easy to interpret and easy to act on.
Until now, the most commonly used metrics for conversational AI provided an incomplete view of assistant success rate. Measures like containment, time to resolution, and coverage failed to comprehensively convey if the assistant was successful in satisfying user requests and, in some cases, were susceptible to gaming.* The conversational AI space was crying out for a more accurate schema for data analysis that would provide content authors the precise data they needed to improve their assistant.
Watson Assistant’s new experience has filled that void with two simple, powerful metrics: Completion and Recognition. Recognition measures how often your assistant successfully funnels a user into an action. Completion measures how often your assistant successfully guide users to the end of an action.
Those two metrics give you the sky-level view of your assistant’s performance, but the redesign of the Analyze page doesn’t stop there; content authors can now review individual conversations to see where the assistant is succeeding and where its conversations need to be refined.
A clear path to improvement
Let’s see an example of a content author using the Analyze page to improve their assistant. Consider a bank that’s built a virtual assistant to help with customer inquiries, handle simple transactions, and generally take the pressure off of their call centers. After their virtual assistant is deployed, the copywriter that built the assistant dives into the analytics to see if there are any conversations where users are getting frustrated or failing to reach the end of conversations.
The assistant’s Recognition numbers are strong, so the copywriter doesn’t need to investigate too thoroughly there, but their Completion metric is showing 17 out of 42 actions were not completed.
Clearly end users are getting stuck somewhere, so she investigates by scrolling down and clicking Action completion. The data displayed on that page breaks down how often actions are being triggered, the total number of incomplete actions, and the completion rate.
‘Withdraw money’ is clearly a problem area, so she clicks on it to examine where and how the user is getting stuck. Of the 10 incomplete actions, 7 were the result of the user getting stuck on a step. The copywriter filters the view on Stuck on step and then reviews the logs of conversations that failed for that reason. The copywriter starts reading through the conversation logs and a pattern emerges: users are typing in ‘debit’ rather than selecting one of the prompts the assistant is offering them.
Our copywriter scrolls up and clicks on the edit tab, which allows content authors to jump straight into the Actions Editor.
She clicks on the step where users are getting stuck, selects Edit response, sees that they currently don’t have any synonyms for checking, and adds ‘debit’.
From there, she simply saves and publishes her changes. Going forward, any time a user types in debit, the assistant will recognize it as a synonym for ‘checking’ and guide the user through the rest of the steps accordingly. Analyze has provided our copywriter the exact information she needs to meaningfully improve her assistant, and she was able to achieve that improvement in a handful of steps!
Smart numbers, smart view
If you’ve been keeping up with the latest and greatest changes and additions to Watson Assistant, you know that the product is divided into two environments: Draft and Live. Analyze separates the data for the two environments across all of its subsections: Overview, Action Completion, and Conversations. This allows you see if test users are finding success with your draft content, then switch to the live environment to investigate end users’ success with your published content. You can also filter the data on a custom date range, a necessary feature since you’ll want to review analytics for your most recently published versions without data from previous versions bleeding in.
You saw the Action completion button in our hypothetical bank example; when you switch to Recognition, there’s an identical button that allows you to view Unrecognized requests.
Rounding out the Overview page are three graphs that give you a quick view of your users’ requests and your assistant’s ability to interpret those requests: Most Frequent Actions, Least Frequent Actions, and Least Completed Actions. With this data in hand, content authors have a comprehensive view of where and when the assistant needs additional training and conversational refinement.
Of course, as we saw in our hypothetical example, numbers alone can’t tell the whole story. To really improve your assistant, you need to see what its conversations look like.
More than numbers
Our bank use case above gave you a solid overview of the Action Completion page. Conversations (accessible via the tab underneath Action Completion on the Overview page) allows you access to the full history of conversations in the selected time period, organized by time, action, and user request.
This page represents another breakthrough in Watson Assistant’s new information architecture. Previous iterations of virtual assistants were only able to show individual messages between users and the assistant. With Actions and the new Watson Assistant IA, messages are now grouped by conversations, allowing you to view the entire back and forth between users and your assistant.
Within the conversation logs are annotations that will help deepen your understanding of the interaction between users and your assistant. When you open up the panel to review conversations, you can see markers indicating when an action is triggered, when a conversation is completed, and when actions are failing. In our hypothetical use case, interactions were marked incomplete when users were repeatedly typing in debit rather than selecting one of the response options.
This format makes it easy for you to customize what conversations you can see. The smart search filters at the top of the page allow you to filter on individual actions and system topics, as well as recognized and unrecognized requests. Once your filters are applied, the table will display every conversation that matches your criteria with the conversation, action and request all listed.
Data that takes you somewhere
Data only matters when it leads to results. That’s why Watson Assistant makes it easy and intuitive to navigate from Analyze to the Actions Editor. When authors see their content needs editing, the process to refine and improve that content is intuitive, frictionless, and fast.
Watson Assistant’s mission is to make it easier than ever for non-technical users to build, publish and improve a virtual assistant. That process isn’t a linear progression but rather a continuous loop. Navigating that loop requires a map that’s both sophisticated and user friendly, and with Analyze, Watson Assistant has provided users with everything they need to train their assistant to be the best version of itself.
If you’re a new user of Watson Assistant, refer to our Getting Started series on getting your first assistant off the ground. If you’ve already built and published an assistant, visit the Analyze page early and often; it’s the most effective tool in your toolbox for identifying and filling in any gaps in understanding between your users and your assistant.
*To get a deeper dive on all the pros and cons of traditional conversational AI metrics, check out Conversational AI: Chatbots that Work by Andrew Freed.
