Botium in a Nutshell, Part 2: Writing Chatbot Tests

Florian Treml
Nov 9, 2018 · 8 min read

This is part 2 of the Botium in a Nutshell series of articles.

While the first part of this series gave an overview of the Botium Stack, this part deals with the daily business in software testing and automation, writing test cases.

Hello, World!

The most basic test case in Botium consists of

  1. submitting a phrase possibly entered by a real user to the chatbot
  2. checking the response of the chatbot with the expected outcome

In Botium, the test cases are described by dialog flows the chatbot is supposed to follow. For a sample “greeting” scenario, the Botium test case looks like this — also known as “BotiumScript”:

#me
hello bot!
#bot
Hello, meat bag! How can I help you ?

BotiumScript is an easy-to-use scripting language, which is interpreted by Botium Core in a virtual machine. Don’t worry if you didn’t understand a word of this sentence, it doesn’t really matter from a user perspective.

You can write BotiumScript as:

  • plain text file with Notepad or any other text editor
  • Excel file
  • CSV file (comma separated values)
  • Javascript Code

For technical details please consult the Botium Wiki.

Convos and Utterances

So, let’s elaborate the “Hello, World!”-example from above. While some users will say “hello”, others maybe prefer “hi”:

#me
hi bot!
#bot
Hello, meat bag! How can I help you ?

Another user may enter the conversation with “hey dude!”

#me
hey dude
#bot
Hello, meat bag! How can I help you ?

And there are plenty of other phrases we can think of. For this most simple use case, there are now at least three or more BotiumScripts to write. So let’s rewrite it.

We name this file hello.convo.txt:

TC01 - Greeting#me
HELLO_UTT
#bot
Hello, meat bag! How can I help you ?

You may have noticed the additional lines at the beginning of the BotiumScript.
The first line contains a reference name for the test case to make it easier for you to locate the failing conversation within your test case library.

And we add another file hello_utt.utterances.txt:

HELLO_UTT
hello bot!
hi bot!
hey dude
good evening
hey are you here
anyone at home ?

The first BotiumScript is a convo file — it holds the structure of the conversation you expect the chatbot to follow.

The second BotiumScript is an utterances file — it holds several phrases for greeting someone, and you expect your chatbot to be able to recognize every single one of them as a nice greeting from the user.

Botium will take care that the convo and utterances files are combined to verify every response of your chatbot to every greeting phrase.

So now let’s assume that your chatbot uses several phrases for greeting the user back. In the morning it is:

#me
HELLO_UTT
#bot
Good morning, meat bag! How can I help you this early ?

And in the evening it is:

#me
HELLO_UTT
#bot
Good evening, meat bag! How can I help you at this late hour ?

Let’s extract the bot responses to another utterances file:

BOT_GREETING_UTT
Good evening
Good morning
Hello
Hi

And now comes the magic, we change the convo file to:

#me
HELLO_UTT
#bot
BOT_GREETING_UTT

Utterances files can be used to verify chatbot responses as well. To summarize:

  • An utterance referenced in a #me-section means: Botium, send every single phrase to the chatbot and check the response
  • An utterance referenced in a #bot-section means: Botium, my chatbot may use any of these answers, all of them are fine

Where to place BotiumScript files ?

The BotiumScript files can be placed in any folder structure you consider to be meaningful for your project. Botium will just scan the base directory recursivly and find all of the files holding convos and utterances.

You can even mix plain text files, CSV files and Excel files within a test suite.

If you got so far and understood everything, you are pretty much ready for using Botium. The rest of this article points to some advanced features, but the usage of convos and utterances is the main concept behind BotiumScript.

Even when using Botium Box, you have to understand BotiumScript — Botium Box provides assistance when writing your BotiumScript test cases, but for most efficient use you really have to understand it.

Using Excel Instead of Text Files

You can use Excel files (*.xlsx) instead of plain text files for describing your convos and utterances. Take this conversation as example:

First Excel sample

You can download the Excel file to use as a template here.

The A-column corresponds to the #me tag, the B-column corresponds to the #bot tag. For adding more than one convo to a worksheet, just leave one row empty:

Second Excel sample

You can use utterances here as well (and everything else supported by text files):

Using utterances in Excel (Convo)
Using utterances in Excel (Utterances)

All you have to tell Botium is:

  • The sheet names where to look for convos
  • The sheet names where to look for utterances
  • The row and column index where to start looking

Please note: the header rows (in the screenshots above) are not relevant to Botium, they are just for the sake of visual appearance. By default, Botium starts to look for convos and utterances at row number 2 and column 1.

You can spread your convos over as many sheets as you want (and we recommend to do so) — see Botium Wiki for details.

On the plus side, using Excel helps in buildling your convo structure in a very intuitive, visual way. On the negative side, code repositories are way better in handling text files than in handling binary files (like Excel).

Using CSV Instead of Text Files

See Botium Wiki.

Advanced BotiumScript Usage

This section highlights some BotiumScript features which may come handy from time to time.

Scripting Memory

BotiumScript can remember parts of the chatbot response for later use. In this example, the order number is reused in a later conversation step:

#me
please send me two salami pizza
#bot
OK. Your order number is $orderNum
#me
What is the status order $orderNum
#bot
Your order will arrive soon

Scripting Memory is disabled by default. See Botium Wiki for details how to activate.

User Interface Elements

Lots of chatbots out there are delivering not only text but also pictures, audio or video to the user. And lots of chatbots speed up the conversation by providing simple user interface elements such as buttons.

#me
please send me two salami pizza
#bot
Please select the size of the pizza
BUTTONS Kids|Normal|Family
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png

The special keywords BUTTONS and MEDIA tell Botium to look out for buttons and media files in the chatbot response.

This concept in BotiumScript is called “asserter”, and you can inject your own custom asserters into BotiumScript (for example, to verify that the chatbot has actually added the order to the order management backend).

Similar behaviour is available on the other end of the conversation as well:

#bot
What Pizza do you want ? Please choose an option.
#me
BUTTON Salami

This will simulate a button click or option selection.

#bot
Did you like your Pizza ?
#me
No. It was ugly.
MEDIA pizza_ugly.png

And this will simulate a picture attachment.

Button click simulation and media attachments are not supported by all Botium connectors. Please consult the documentation of the connector in question. The default behaviour is to use the button title as message text.

For technical details please consult the Botium Wiki.

Splitting Convos

To keep your convo files clean, convos can be splitted and included in other convos. This concept is called partial convo in BotiumScript.

Here is a file partial_login.pconvo.txt:

PARTIAL_LOGIN#bot
Please tell me your customer access code!
#me
1234567
#bot
Thanks, you are logged in now.

And here another one show_info.convo.txt:

SHOW_INFO#me
Show my registration info.
INCLUDE PARTIAL_LOGIN#bot
OK, here is your registration info:

The conversation steps from the partial convo are inserted at the INCLUDE statement (the partial convo is selected by name).

Frequently Asked Questions

Doesn’t it take a long time to write test cases ?

Yes, absolutely, writing test cases takes a lot of time. The Botium Stack gives you some tools to speed up the process:

  • A Capture&Replay tool to capture a conversation with your chatbot and generate a convo file out of it
  • Several importers for generating convo files out of chatbot development tools (such as Dialogflow and IBM Watson Assistant)
  • If you have existing conversation logs, they usually can be converted to convo files with reasonable effort
  • The Botium Platform includes hundred thousand of prebuilt test cases for different domains

My chatbot responses include dynamic or alternating text — how to handle this with BotiumScript ?

Apart from separating your testcases in convo and utterances files, it is possible to fine tune how Botium actually verifies the chatbot responses:

  • fixed matching
  • lowercase matching
  • substring matching
  • regular expression matching
  • or you can even implement your own logic and hook it into Botium

For technical details please consult the Botium Wiki.

Can I use BotiumScript for training my chatbot, and how ?

Absolutely. Just send all of your convos to the chatbot and the training begins. How this is actually done depends on the chatbot technology you have choosen. When doing it with IBM Watson Assistant for example, in the Watson Workspace you get a list of the utterances the chatbot did not understand and you are supposed to train them for intents or entities.

For some providers there even are converters available to convert BotiumScript utterances file to the native intent resolution format of the provider.

Can I add the BotiumScript files to a source code repository like git ?

Yes, and we encourage you to do so.

I wrote a lot of BotiumScript files, how can I finally send them to my chatbot ?

See next part of this series.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

https://github.com/codeforequity-at/botium-core

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store