Introducing the Conversation Optimizer for IBM Watson

Niklas Heidloff
Aug 29, 2017 · 3 min read

In order to develop conversational experiences, it’s important to understand how users are actually using bots. Based on this data, conversational experiences can be improved continuously.

Watson Conversation comes with an improve component, which helps with analyzing and improving bots. Depending on the service plan, all conversations from the last 7, 30 or 90 days are stored. You can get an overview of the interactions between users and bots via the Watson Conversation web interface, like the traffic at a specific date, for example. You can also improve how Watson understands user input by adding or changing intents and entities. Additionally, there is a new recommendations feature, which I want to blog about soon.

Watson Conversation provides an API to read the logs. Ansgar Schmidt and I have used this API to built a new tool that helps optimizing conversational experiences. The tool reads conversations of a workspace and provides different filters to find conversation flows that have potential issues and might require improvements.

The Conversation Optimizer for IBM Watson is available as open source. You can install it by clicking the Deploy button or you can run an online demo version. In order to add conversations to the demo version, you can use the Conversation Inspector, which has been configured to go against the same workspace.

Here is a screenshot showing a list of conversations with potential issues using the sample car dashboard workspace:

At this point four filters are provided.

1. Out of Scope

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: Who is Niklas?
Watson: I’m not sure about that. You can say things like “Turn on my lights” or “Play some music.”

In order for this filter to work the dialog needs to be changed by adding ‘”output.outOfScope” : ““‘ to the nodes when the user input is out of scope. These are typically the nodes with the ‘anything_else’ condition.

2. Unfulfilled Intents

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: turn on music
Watson: Sure thing! Which genre would you prefer? Jazz is my personal favorite.
User: Neue Deutsche Welle
Watson: Sorry I don’t recognize that genre. Want to try something? Jazz? Rock?

This filter also requires changes to the dialog definition. Developers can mark intents as being fulfilled when certain nodes in the dialogs are reached. This way intents that have been triggered but not fulfilled can be detected. For this feature to work you need to add information to the JSON of your dialog nodes:
– Add ‘”output.intentTriggered” : “your-intent-name”‘ to the node when the intent is triggered
– Add ‘”output.intentFulfilled” : “your-intent-name”‘ to the node when the intent is fulfilled

3. Unconfident Intents

Sample Conversation
Watson: Hi. It looks like a nice drive today. What would you like me to do?
User: make it warmer
Watson: I understand you want me to turn up something. You can say increase the volume or turn up the AC.

This filter shows all conversations with intents that have less than 50 percent confidence.

4. Dialog Errors

This filter shows conversations with log messages, which indicate errors in the dialog definition.

We’d be happy to get your feedback !

Originally published at heidloff.net on August 29, 2017.

)

Written by

Developer Advocate. I like learning, conversational UX, serverless and BBQ. Father of five. Working for IBM. Opinions are my own.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade