Analyze chatbot classifier performance from logs

Andrew R. Freed
Dec 11, 2019 · 2 min read
Image for post
Image for post
Explore log data to determine how well your chatbot performs. Photo by John Schnobrich on Unsplash

This post is part of a series:
1) Testing a Chatbot with k-folds Cross Validation
2) Analyze chatbot classifier performance from logs
3) Improve a chatbot classifier with production data

In the embedded video I demonstrate reviewing user utterances and how to turn them into “blind test” and future training data with the WA-Testing-Tool.

I first describe how to collect user utterances from logs and how to manually classify them into intents, taking care to make sure each example represents a single clear intent and removes superfluous text. For example “Hi, thanks for helping, can you tell me where the store is?” should be shortened to “Can you tell me where the store is?” and associated with the “Store Location” intent. I also describe techniques for group review including the notion that if a consensus can not be reached on an example in ten seconds, it is not an obvious example and should not be used for training.

The video culminates in the creation of a blind test set and a blind test evaluation through WA-Testing-Tool including an accuracy measurement. Areas for improvement are discussed but will be addressed in the next video.

10 minute video on reviewing log data and analyzing chat performance in a blind test.

For help in implementing these practices, reach out to IBM Data and AI Expert Labs and Learning.

IBM Data and AI

AI Trust | Automation | Language

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store