Botium in a Nutshell, Part 3: Automating Chatbot Tests

Florian Treml
Nov 9, 2018 · 5 min read

This is part 3 of the Botium in a Nutshell series of articles.

Now we are really close to let Botium talk to your chatbot! First we have to tell Botium how to actually connect to your chatbot. Now is the time to install the Botium CLI, the swiss army knife of Botium!

Connecting to your Chatbot

Botium has to somehow build up a communication channel with your chatbot. Usually you have to talk to your development team to provide you the information in a configuration file botium.json — for a chatbot developed with the Microsoft Bot Framework it looks like this:

"botium": {
"Capabilities": {
"PROJECTNAME": "Directline3 Plugin Sample",
"CONTAINERMODE": "directline3",
"DIRECTLINE3_SECRET": "my-directline-secret",

The configuration items in Botium are called capabilities. They are different for each supported chatbot technology. Botium supports a large variety of chatbot technologies — chatbots built with Dialogflow or Microsoft Bot Framework are supported, as well as chatbots publishing a HTTP/JSON API, or chatbots running on a voice-enabled home automation device, or chatbots running within a chat widget on a website.

The one most important capability to tell Botium what technology to use to connect to your chatbot is CONTAINERMODE. In the example above directline3 is given to instruct Botium to use the Microsoft Bot Framework technology called Directline, Version 3, for a chatbot hosted in Azure. Out-of-the-box, there are several connector technologies supported:

  • directline3 for Microsoft Bot Framework Bots hosted on Azure
  • watson for chatbots developed with IBM Watson
  • dialogflow for chatbots developed with Google Dialogflow
  • fbdirect for chatbots deployed to a Facebook page
  • simplerest for chatbots publishing an HTTP/JSON endpoint (Rasa endpoint, or custom endpoints)
  • docker for packaging your chatbot code in a docker environment surrounded by infrastructure mocks (advanced!)
  • alexa-smapi for Alexa skills (with the Skill Management API)
  • alexa-avs for Alexa skills on End-To-End-Level (Text-To-Speech and Speech-To-Text conversion)
  • webdriverio for end-to-end tests on user interface level (chatbots published as website widget)
  • webspeech for using the HTML5 Webspeech API to interact with spoken language (nice for demonstration purposes)
  • And more

Furthermore, with Botiums open interfaces it is possible to plug your own custom chatbot technology into Botium (Node.js developer skills required).

Botium Box includes management of your chatbot capabilities, so you won’t have to write those botium.json files by hand. Instead the configuration is done within an easy 3-step-wizard.

For technical details about capabilities please consult the Botium Wiki.

You can validate your configuration settings by using the Botium CLI — it opens a REPL-interface for you, so you can have a chat with your chatbot:

> botium-cli emulator --config path/to/botium.json

The Botium Box includes a “Live Chat” with the same purpose.

Botium Box Live Chat

Test Case Execution

After following this guide, we now have:

  • a couple of convo and utterances files, a test set
  • a valid set of capabilities for connecting to your chatbot

What can you do with it ?

Option 1: Botium CLI

Run Botium CLI like this:

> botium-cli run --config path/to/botium.json --convos path/to/files

Botium CLI will build up a communication channel with your chatbot and run all of your convo files. Status information and a summary are displayed in the command line window.

Option 2: Botium Bindings

Using Botium Bindings is worth a whole article for itself, so please read on here if interested in using an estabilished test runner like Mocha, Jest or Jasmine for running Botium test cases.

Option 3: Botium Box

Use the Quickstart Wizard to connect your chatbot to your test sets and run them.

1. Configure Chatbot Capabilities
2. Select test sets
3. Start the test session
4. View the test report and inspect failed test cases

Frequently Asked Questions

How can I connect to my chatbot ?

For technical details about supported chatbot technologies and capabilities please consult the Botium Wiki.

My chatbot doesn’t send any response. Please help.

You can find the Troubleshooting Guide in the Botium Wiki. Usually it means that the capabilities in your botium.json are invalid.

Can I integrate Botium with my CI Pipeline from Vendor XYZ / with my costum CI Pipeline?

Yes, absolutely. As long as your CI Pipeline is capable of running a command line script (when using Botium CLI) or calling a HTTP Backend (when using Botium Box), you are fine. Botium Box and Botium CLI support a couple of different test report output formats which usually can be parsed by any pipeline.

I want to see a mega-hipster-responsive-drill-down test report — where is it ?

Botium CLI is not a test report renderer, this task is left to the CI Pipeline and to Botium Box. Nevertheless, Botium CLI supports outputting a mochawsome test report, which may fulfill your needs.

Can Botium send me an E-Mail message when it is ready ?

No. This task is left to the CI Pipeline, or you can write your own shell scripts around Botium CLI to send E-Mails after completion — this is one of the strength of the command line approach, you can use Botium from any shell script. Here is a sample how you can possibly do it.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store