Introducing the Conversation Optimizer for IBM Watson

In order to develop conversational experiences, it’s important to understand how users are actually using bots. Based on this data, conversational experiences can be improved continuously.

Watson Conversation comes with an improve component, which helps with analyzing and improving bots. Depending on the service plan, all conversations from the last 7, 30 or 90 days are stored. You can get an overview of the interactions between users and bots via the Watson Conversation web interface, like the traffic at a specific date, for example. You can also improve how Watson understands user input by adding or changing intents and entities. Additionally, there is a new recommendations feature, which I want to blog about soon.

Watson Conversation provides an API to read the logs. Ansgar Schmidt and I have used this API to built a new tool that helps optimizing conversational experiences. The tool reads conversations of a workspace and provides different filters to find conversation flows that have potential issues and might require improvements.

The Conversation Optimizer for IBM Watson is available as open source. You can install it by clicking the Deploy button or you can run an online demo version. In order to add conversations to the demo version, you can use the Conversation Inspector, which has been configured to go against the same workspace.

Here is a screenshot showing a list of conversations with potential issues using the sample car dashboard workspace:

At this point four filters are provided.

1. Out of Scope

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: Who is Niklas?
Watson: I’m not sure about that. You can say things like “Turn on my lights” or “Play some music.”

In order for this filter to work the dialog needs to be changed by adding ‘”output.outOfScope” : ““‘ to the nodes when the user input is out of scope. These are typically the nodes with the ‘anything_else’ condition.

2. Unfulfilled Intents

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: turn on music
Watson: Sure thing! Which genre would you prefer? Jazz is my personal favorite.
User: Neue Deutsche Welle
Watson: Sorry I don’t recognize that genre. Want to try something? Jazz? Rock?

This filter also requires changes to the dialog definition. Developers can mark intents as being fulfilled when certain nodes in the dialogs are reached. This way intents that have been triggered but not fulfilled can be detected. For this feature to work you need to add information to the JSON of your dialog nodes:
– Add ‘”output.intentTriggered” : “your-intent-name”‘ to the node when the intent is triggered
– Add ‘”output.intentFulfilled” : “your-intent-name”‘ to the node when the intent is fulfilled

3. Unconfident Intents

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: make it warmer
Watson: I understand you want me to turn up something. You can say increase the volume or turn up the AC.

This filter shows all conversations with intents that have less than 50 percent confidence.

4. Dialog Errors

This filter shows conversations with log messages, which indicate errors in the dialog definition.

We’d be happy to get your feedback !

Originally published at heidloff.net on August 29, 2017.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.