Do You Know Where and Why Users Drop Off the Conversation?

Use the new Dialog Flow Analysis Notebook to understand where and why users abandon your Virtual Assistant

Co-Authored by Avi Yaeli and Eric Wayne.

DDo you feel blind and overwhelmed by the task of improving your virtual assistant? Try our new Dialog Flow Analysis Notebook. You’ll identify problems that keep your users from being successful.

Photo by giovanni sciacca on Unsplash

You have a purpose in mind with each assistant that you create. Are you providing information or helping your users perform a task? Delivering an assistant requires training to detect what the user is trying to achieve (intents and entities). You must define how to help the user complete a task (dialog flow). Then you need to monitor your assistant to detect any problems your users have. Watson Assistant includes powerful features to reduce the chance of your users getting stuck. These features include branching, jump-to, conditional responses, and digression support. When you combine this with large amounts of usage data, the task of finding problems can be overwhelming. You need help finding patterns of problems.

You want to be sure users are not getting stuck at a specific step or abandoning the conversation altogether.

A standard technique for analyzing abandonment is to perform a manual assessment of user conversation logs. As you read the text of conversations, you can try to understand the flow of dialog nodes with the hope of finding problematic patterns. For the owner of a virtual assistant, this can be very time consuming, and you may still overlook patterns.

This is why we just released the Dialog Flow Analysis Notebook. This Python notebook can help you analyze the dialog flow of low-quality conversations. The notebook provides analysis and interactive visualizations that can help you with questions such as:

• What are the actual conversation steps and flow ratios within the assistant?
• Which flows have high volume but low completion rates (ineffective conversation flows)?
• Where along the dialog steps users lose engagement with your assistant?
• What are the potential dialog steps or terms in user utterances that may lead to abandonment?
• How do flows change after rolling out a new version into production

Example

Imagine you are a financial service provider. Your assistant is helping clients schedule an appointment with an investment specialist at their location of choice. While measuring the performance of your assistant you observe that only 32% of the users who start are able to complete the task and schedule an appointment.

Measuring task completion rates. The Schedule Appointment task has 32% completion rate

At that point you want to understand why users weren’t able to complete the task and if there’s anything you can do to improve the completion rate. There are thousands of conversations handled by that task every day and sifting through logs is tedious and doesn’t scale. You need a more effective way to summarize the journey of steps a user takes in the task and find common failure patterns.

The dialog flow visualization

The dialog flow visualization is an interactive tool for investigating user journeys, visits, and abandonments within the steps of the dialog logic. You can aggregate the steps across the assistant and by specific user segments or time periods. You can use the visualization to understand a task flow from start to finish or focus your exploration on key steps.

The Dialog Flow Visualization — showing the distribution of the main user flows at the beginning of the conversation with the assistant.

The visualization is interactive, allowing you to drill down into steps and observe statistics such as the number of visits and the rates of abandonment. The dialog flow analysis and visualization components provide a rich set of options that you can use to analyze subsets of journeys, journeys across selected points of interests, reverse journeys, and trends.

The Dialog Flow Visualization — showing the trend of reverse journeys that end with escalation across two consecutive weeks.

Analyzing abandoned conversations

Observing failure patterns such as user drop off at an unexpected location in the dialog can help you identify where users disengage. When you click on a node in the visualization, you are also selecting the group of abandoned conversations. Then you can analyze only this group of conversations.

A milestone-level view of user flows through the dialog logic. The conversations that abandoned at the ‘Branch selection’ node are selected for further analysis and assessment

The next step requires you to drill down into user utterances and conversation steps to understand what might have led to the users’ drop off. Some possible reasons could be:

• The assistant didn’t expect or understand the user at that point of the conversation
• The options presented to the user were not relevant
• The conversation didn’t progress in the right direction
• The assistant’s response wasn’t accurate or relevant

The Dialog Flow Analysis Notebook helps you investigate those conversations using two visual components:

• The transcript analyzer shows the user utterances, the assistant response, and the dialog nodes used to produce the response. This can help you to quickly assess and pinpoint potential issues related to the dialog progression.

The transcript analyzer — showing every step of the user journey and which dialog nodes were triggered to produce a response. User utterances with negative sentiment are highlighted in order to provide a quick starting place for investigating issues.

In the above example, the user dropped off at the Branch Selection step because the map of branches shown in the previous step did not include nearby branches. Also, the assistant did not understand her negative feedback at that point in the dialog. The next improvement might be to train the assistant to respond to similar utterances at this point of the dialog. Or you might check the quality of the branch locating mapping service.

• The keywords and phrases analysis let you discover common terms that are frequent and representative within the group of selected conversations. This can help you determine if you need to train new or update existing intents or entities. You may also decide you need to revise the implementation of a dialog node.

Example output of the keywords and phrases highlighting common terns in abandoned conversations

Staying with the same example from the transcript analyzer, keywords and phrases can also highlight areas for improvement. You can see keywords such as wrong, location, map, and phrases such as wrong branches, map error appear frequently in the conversations that drop off at the Branch Selection step.

The notebook concludes by providing a few possible next steps. You can export candidate user utterances for Watson Intent Recommendation. Or you can export a sample of conversations for detailed analysis using the Effectiveness Notebook.

In Conclusion

The Dialog Flow Analysis Notebook introduces new visual analytics to help you discover and investigate problems with your assistant. This notebook extends and integrates with the Measure and Analyze notebooks. The IBM Watson Assistant Continuous Improvement Best Practices guide describes how to use these notebooks together. We would love to get feedback on your experience using Watson Assistant and the new Dialog Flow Analysis notebook.

Getting Started

You can fork or download the Dialog Flow Analysis Notebook from Github: https://github.com/watson-developer-cloud/assistant-dialog-flow-analysis.

Check out the Getting Started section on how to run using Jupyter Notebook or Watson Studio. The repository also includes a sample log dataset which you can use to quickly assess how the notebook works, before connecting it to your own assistant logs.

For those of you who do not want to worry about the installation process, we have also made available a hosted version of the notebook on the IBM Gallery.

If you have any issues or questions, please create issues on the Github repository and we will try and address them as soon as we can.

Last but not least…

Special thanks to Sergey Zeltyn and Zhe Zhang for their contribution to the development of the notebook and the underlying analysis library.