Tutorial: Using Paraphrasing to Increase Conversational AI Test Coverage

Florian Treml
Feb 27 · 5 min read

UPDATE 2020/06/15: As Chatbots grow in importance, automated testing solutions will remain critical for ensuring that Chatbots actually do what their designers intend. We’ve been busy working on a product that allows testers to have visual insights and deeper understanding in their Chatbot’s performance, offering several solutions to boost their interaction!

Botium Coach will be introduced to the market as part of our online event on the 24th of June.

Stay tuned with our teaser video and register for our product launch!

This tutorial provides a step-by-step introduction how to use Botium Box to quickly build a regression test suite for an Amazon Lex chatbot and increase the test coverage by adding paraphrases to the test data.

What we will do:

  1. Create an Amazon Lex chatbot and enable programmatic access
  2. Connect Botium Box to Lex
  3. Automatically generate basic test cases in Botium Box out of the Lex model
  4. Use the Botium Box Paraphraser to augment the test cases
  5. View the results of a test run

If you don’t know what Botium is about, you should first read about it in the Botium in a Nutshell blog series.

1. Create an Amazon Lex Chatbot

Follow the instructions in the official Lex documentation to create a bot using a blueprint (OrderFlowers): https://docs.aws.amazon.com/lex/latest/dg/gs-bp-create-bot.htmlYou only need step 1 for this tutorial.

Image for post
Image for post
Amazon Lex Console

Afterwards, publish a first version of the bot: https://docs.aws.amazon.com/lex/latest/dg/gettingstarted-ex3.html

Now create an IAM user and make sure to enable programmatic access. You will need the access key and the secret later.

Image for post
Image for post
Programmatic access

Choose Attach existing policies to user directly to give permissions AmazonLexFullAccess

Image for post
Image for post
AmazonLexFullAccess policy

2. Connect Botium Box to Lex

For this tutorial, it is assumed that you already have a working installation of Botium Box.

Get Botium Box Community Edition here!

Image for post
Image for post
Botium Box Dashboard

Register a new chatbot (don’t forget to save):

  • Select Amazon Lex as Connector/Chatbot Technology
  • Enter the IAM Access Key and IAM Secret Key from previous step
  • Select the Amazon Region Code, the name of the bot and the alias you used for publishing
Image for post
Image for post
Chatbot Registration

Use Say Hello to check your credentials. Your blueprint OrderFlowers bot should answer with something like I didn’t understand.

Image for post
Image for post
Botium Box Say Hello

You can also start a quick Live Chat with your Lex bot.

Image for post
Image for post
Botium Box Live Chat

3. Generate Test Cases

Now that the infrastructure is set, we are making Botium Box do the hard work for us and let it automatically generate the boilerplate test cases from the Lex model. Click the Import Lex Model as Test Cases button, select a test set name and click OK. Within a few seconds, Botium Box

  • downloads the Lex model
  • analyzes the structure of the intents and entities
  • and generates test cases of all known utterances with intent and entity asserters
Image for post
Image for post
Import Lex Model as Test Cases

The boilerplate Lex bot only has one intent with two user examples, so the generated test case is rather thin.

Image for post
Image for post
OrderFlowers utterance file

4. Data Augmentation

Clicking on Paraphrase It! will make Botium Box suggest additional user examples based on the two existing ones already defined in the Lex model.

Image for post
Image for post
Getting Paraphrases

So the Lex model originally included only two user examples:

I would like to pick up flowers
I would like to order some flowers

After using Paraphrase It! we now have eight utterances with more or less the same meaning — your Lex bot should definitely be able to predict the right intent OrderFlowers for all of these user examples:

I would like to pick up flowers
I would like to order some flowers
I'd like to pick flowers
I'd like to take flowers
I want to pick flowers
I'd like to order flowers
I'd like to order some flowers
I want to order some flowers

5. Test Run

Create a Botium Box Test Project with the chatbot you registered and the test set and let it run. In my case, Lex was unable to predict the correct intent for the user example I’d like to take flowers — all other user examples worked.

Image for post
Image for post
Botium Box Test Session

It’s now up to you to decide what to do with this information — train the Lex bot with additional user examples, or remove it from the test set.

Conclusion

With the help of Botium Box it is possible to create a regression test suite for a chatbot and semi-automatically augment the test data within several minutes. You can give it a try immediately.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store