Botium in a Nutshell, Part 2: Writing Chatbot Tests

This is part 2 of the Botium in a Nutshell series of articles.

While the first part of this series gave an overview of the Botium Stack, this part deals with the daily business in software testing and automation, writing test cases.

Hello, World!

The most basic test case in Botium consists of

  1. submitting a phrase possibly entered by a real user to the chatbot
  2. checking the response of the chatbot with the expected outcome

In Botium, the test cases are described by dialog flows the chatbot is supposed to follow. For a sample “greeting” scenario, the Botium test case looks like this — also known as “BotiumScript”:

#me
hello bot!
#bot
Hello, meat bag! How can I help you ?
BotiumScript is an easy-to-use scripting language, which is interpreted by Botium Core in a virtual machine. Don’t worry if you didn’t understand a word of this sentence, it doesn’t really matter from a user perspective.

You can write BotiumScript as:

  • plain text file with Notepad or any other text editor
  • Excel file
  • CSV file (comma separated values)
  • Javascript Code

For technical details please consult the Botium Wiki.

Convos and Utterances

So, let’s elaborate the “Hello, World!”-example from above. While some users will say “hello”, others maybe prefer “hi”:

#me
hi bot!
#bot
Hello, meat bag! How can I help you ?

Another user may enter the conversation with “hey dude!”

#me
hey dude
#bot
Hello, meat bag! How can I help you ?

And there are plenty of other phrases we can think of. For this most simple use case, there are now at least three or more BotiumScripts to write. So let’s rewrite it.

We name this file hello.convo.txt:

TC01 - Greeting
#me
HELLO_UTT
#bot
Hello, meat bag! How can I help you ?
You may have noticed the additional lines at the beginning of the BotiumScript.
The first line contains a reference name for the test case to make it easier for you to locate the failing conversation within your test case library.

And we add another file hello_utt.utterances.txt:

HELLO_UTT
hello bot!
hi bot!
hey dude
good evening
hey are you here
anyone at home ?

The first BotiumScript is a convo file — it holds the structure of the conversation you expect the chatbot to follow.

The second BotiumScript is an utterances file — it holds several phrases for greeting someone, and you expect your chatbot to be able to recognize every single one of them as a nice greeting from the user.

Botium will take care that the convo and utterances files are combined to verify every response of your chatbot to every greeting phrase.

So now let’s assume that your chatbot uses several phrases for greeting the user back. In the morning it is:

#me
HELLO_UTT
#bot
Good morning, meat bag! How can I help you this early ?

And in the evening it is:

#me
HELLO_UTT
#bot
Good evening, meat bag! How can I help you at this late hour ?

Let’s extract the bot responses to another utterances file:

BOT_GREETING_UTT
Good evening
Good morning
Hello
Hi

And now comes the magic, we change the convo file to:

#me
HELLO_UTT
#bot
BOT_GREETING_UTT

Utterances files can be used to verify chatbot responses as well. To summarize:

  • An utterance referenced in a #me-section means: Botium, send every single phrase to the chatbot and check the response
  • An utterance referenced in a #bot-section means: Botium, my chatbot may use any of these answers, all of them are fine

Where to place BotiumScript files ?

The BotiumScript files can be placed in any folder structure you consider to be meaningful for your project. Botium will just scan the base directory recursivly and find all of the files holding convos and utterances.

You can even mix plain text files, CSV files and Excel files within a test suite.

If you got so far and understood everything, you are pretty much ready for using Botium. The rest of this article points to some advanced features, but the usage of convos and utterances is the main concept behind BotiumScript.

Even when using Botium Box, you have to understand BotiumScript — Botium Box provides assistance when writing your BotiumScript test cases, but for most efficient use you really have to understand it.

Advanced BotiumScript Usage

This section highlights some BotiumScript features which may come handy from time to time.

Scripting Memory

BotiumScript can remember parts of the chatbot response for later use. In this example, the order number is reused in a later conversation step:

#me
please send me two salami pizza
#bot
OK. Your order number is $orderNum
#me
What is the status order $orderNum
#bot
Your order will arrive soon

User Interface Elements

Lots of chatbots out there are delivering not only text but also pictures, audio or video to the user. And lots of chatbots speed up the conversation by providing simple user interface elements such as buttons.

#me
please send me two salami pizza
#bot
Please select the size of the pizza
BUTTONS Kids|Normal|Family
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png

The special keywords BUTTONS and MEDIA tell Botium to look out for buttons and media files in the chatbot response.

This concept in BotiumScript is called “asserter”, and you can inject your own custom asserters into BotiumScript (for example, to verify that the chatbot has actually added the order to the order management backend).

For technical details please consult the Botium Wiki.

Frequently Asked Questions

Doesn’t it take a long time to write test cases ?

Yes, absolutely, writing test cases takes a lot of time. The Botium Stack gives you some tools to speed up the process:

  • A Capture&Replay tool to capture a conversation with your chatbot and generate a convo file out of it
  • Several importers for generating convo files out of chatbot development tools (such as Dialogflow and IBM Watson Assistant)
  • If you have existing conversation logs, they usually can be converted to convo files with reasonable effort
  • The Botium Platform includes hundred thousand of prebuilt test cases for different domains

My chatbot responses include dynamic or alternating text — how to handle this with BotiumScript ?

Apart from separating your testcases in convo and utterances files, it is possible to fine tune how Botium actually verifies the chatbot responses:

  • fixed matching
  • lowercase matching
  • substring matching
  • regular expression matching
  • or you can even implement your own logic and hook it into Botium

For technical details please consult the Botium Wiki.

Can I use BotiumScript for training my chatbot, and how ?

Absolutely. Just send all of your convos to the chatbot and the training begins. How this is actually done depends on the chatbot technology you have choosen. When doing it with IBM Watson Assistant for example, in the Watson Workspace you get a list of the utterances the chatbot did not understand and you are supposed to train them for intents or entities.

For some providers there even are converters available to convert BotiumScript utterances file to the native intent resolution format of the provider.

Can I add the BotiumScript files to a source code repository like git ?

Yes, and we encourage you to do so.

I wrote a lot of BotiumScript files, how can I finally send them to my chatbot ?

See next part of this series.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here: