Do You Know Where and Why Users Drop Off the Conversation?
Use the new Dialog Flow Analysis Notebook to understand where and why users abandon your Virtual Assistant
Co-Authored by Avi Yaeli and Eric Wayne.
DDo you feel blind and overwhelmed by the task of improving your virtual assistant? Try our new Dialog Flow Analysis Notebook. You’ll identify problems that keep your users from being successful.
You have a purpose in mind with each assistant that you create. Are you providing information or helping your users perform a task? Delivering an assistant requires training to detect what the user is trying to achieve (intents and entities). You must define how to help the user complete a task (dialog flow). Then you need to monitor your assistant to detect any problems your users have. Watson Assistant includes powerful features to reduce the chance of your users getting stuck. These features include branching, jump-to, conditional responses, and digression support. When you combine this with large amounts of usage data, the task of finding problems can be overwhelming. You need help finding patterns of problems.
You want to be sure users are not getting stuck at a specific step or abandoning the conversation altogether.
A standard technique for analyzing abandonment is to perform a manual assessment of user conversation logs. As you read the text of conversations, you can try to understand the flow of dialog nodes with the hope of finding problematic patterns. For the owner of a virtual assistant, this can be very time consuming, and you may still overlook patterns.
This is why we just released the Dialog Flow Analysis Notebook. This Python notebook can help you analyze the dialog flow of low-quality conversations. The notebook provides analysis and interactive visualizations that can help you with questions such as:
- What are the actual conversation steps and flow ratios within the assistant?
- Which flows have high volume but low completion rates (ineffective conversation flows)?
- Where along the dialog steps users lose engagement with your assistant?
- What are the potential dialog steps or terms in user utterances that may lead to abandonment?
- How do flows change after rolling out a new version into production
Example
Imagine you are a financial service provider. Your assistant is helping clients schedule an appointment with an investment specialist at their location of choice. While measuring the performance of your assistant you observe that only 32% of the users who start are able to complete the task and schedule an appointment.
At that point you want to understand why users weren’t able to complete the task and if there’s anything you can do to improve the completion rate. There are thousands of conversations handled by that task every day and sifting through logs is tedious and doesn’t scale. You need a more effective way to summarize the journey of steps a user takes in the task and find common failure patterns.
The dialog flow visualization
The dialog flow visualization is an interactive tool for investigating user journeys, visits, and abandonments within the steps of the dialog logic. You can aggregate the steps across the assistant and by specific user segments or time periods. You can use the visualization to understand a task flow from start to finish or focus your exploration on key steps.
The visualization is interactive, allowing you to drill down into steps and observe statistics such as the number of visits and the rates of abandonment. The dialog flow analysis and visualization components provide a rich set of options that you can use to analyze subsets of journeys, journeys across selected points of interests, reverse journeys, and trends.
Analyzing abandoned conversations
Observing failure patterns such as user drop off at an unexpected location in the dialog can help you identify where users disengage. When you click on a node in the visualization, you are also selecting the group of abandoned conversations. Then you can analyze only this group of conversations.
The next step requires you to drill down into user utterances and conversation steps to understand what might have led to the users’ drop off. Some possible reasons could be:
- The assistant didn’t expect or understand the user at that point of the conversation
- The options presented to the user were not relevant
- The conversation didn’t progress in the right direction
- The assistant’s response wasn’t accurate or relevant
The Dialog Flow Analysis Notebook helps you investigate those conversations using two visual components:
- The transcript analyzer shows the user utterances, the assistant response, and the dialog nodes used to produce the response. This can help you to quickly assess and pinpoint potential issues related to the dialog progression.
In the above example, the user dropped off at the Branch Selection step because the map of branches shown in the previous step did not include nearby branches. Also, the assistant did not understand her negative feedback at that point in the dialog. The next improvement might be to train the assistant to respond to similar utterances at this point of the dialog. Or you might check the quality of the branch locating mapping service.
- The keywords and phrases analysis let you discover common terms that are frequent and representative within the group of selected conversations. This can help you determine if you need to train new or update existing intents or entities. You may also decide you need to revise the implementation of a dialog node.
Staying with the same example from the transcript analyzer, keywords and phrases can also highlight areas for improvement. You can see keywords such as wrong, location, map, and phrases such as wrong branches, map error appear frequently in the conversations that drop off at the Branch Selection step.
The notebook concludes by providing a few possible next steps. You can export candidate user utterances for Watson Intent Recommendation. Or you can export a sample of conversations for detailed analysis using the Effectiveness Notebook.
In Conclusion
The Dialog Flow Analysis Notebook introduces new visual analytics to help you discover and investigate problems with your assistant. This notebook extends and integrates with the Measure and Analyze notebooks. The IBM Watson Assistant Continuous Improvement Best Practices guide describes how to use these notebooks together. We would love to get feedback on your experience using Watson Assistant and the new Dialog Flow Analysis notebook.
Getting Started
You can fork or download the Dialog Flow Analysis Notebook from Github: https://github.com/watson-developer-cloud/assistant-dialog-flow-analysis.
Check out the Getting Started section on how to run using Jupyter Notebook or Watson Studio. The repository also includes a sample log dataset which you can use to quickly assess how the notebook works, before connecting it to your own assistant logs.
For those of you who do not want to worry about the installation process, we have also made available a hosted version of the notebook on the IBM Gallery.
If you have any issues or questions, please create issues on the Github repository and we will try and address them as soon as we can.
Last but not least…
Special thanks to Sergey Zeltyn and Zhe Zhang for their contribution to the development of the notebook and the underlying analysis library.