Botium in a Nutshell, Part 5: The Botium Box

Florian Treml
Jan 19, 2019 · 5 min read

This is part 5 of the Botium in a Nutshell series of articles.

This article is an introduction for Botium Box users. There is a trial and a free community edition available of the Botium Box. The best way to explore Botium Box is to start with the trial, please register here.

Botium Box Taxonomy

As a starter a short overview of the terms used in this article an in the Botium Box:

  • Test Cases are written as Convos and Utterances — see here
  • A Test Set is a collection of Test Cases
  • A chatbot in Botium Box is a set of capabilities to tell Botium how to connect to a single instance of a chatbot
  • A test project is a combination of test sets and a chatbot — the specification of a test environment
  • A test session is the process of sending test sets to a chatbot and asserting the correctness of the results
  • A device set is a collection of device or browser specifications to be used for user interface testing
  • An agent is a piece of software for test execution. The more agents are connected to the Botium Box, the more tests can run in parallel.

The Dashboard

The Dashboard is the entry point. You can see an overview how the quality of your chatbot evolves over time, how the chatbot performs in your environments and what test sets perform well.

Botium Box Dashboard

Connecting your chatbot

Botium supports a large variety of chatbot technologies. For each technology, the configuration form contains inline help how to retrieve the required connection information, tokens, secrets etc.

Chatbot capability configuration screen

You can always switch to the Advanced Settings mode to get fine tune the capabilities.

Live Chat

For checking your configuration, you can always start a live chat with your chatbot.

Starting the live chat

Just click CONNECT to start Botium Core in the background and talk to your chatbot.

Live Chat screen

You can save the conversation as test case. This is the quick way to start buildling your custom test set.

Save conversation as Test Case

The Test Set Library and Humanification

Test case files are managed in test set libraries. Within a single test set, you can combine test cases from various sources:

  • Write Convos and Utterances in Botium Box
  • Upload Excel files
  • Register a shared folder holding the test cases
  • Register a Git repository holding the test cases

Note: in real-life scenarios, the last option (Git repository) is clearly the best option as it allows the test cases to be unter version control.

Test Set view

When cloning a test set, you can optionally apply various humanification algorithms: the test set is then modified to take human habbits into account and to appear more like human input to the chatbot:

  • space and newline handling
  • sausage fingers
  • common typographic errors for qwertz/qwerty keyboards
  • emojis
  • mishits
  • case sensitivity
  • and many more …
Humanification algorithms

Starting a Test Session

On the Test Results screen, a test session can be started. Depending on the number of test agents, the performance of the chatbot and the number of testcases this will take some time.

Starting a test session

Exploring the Test Results

The test report shows on a very detailed level every aspect of the test execution:

  • Successful and failed test cases
  • Transcripts for all executed test cases
  • Detailed log files for all test execution jobs
  • Performance measurement
Test Results

The test results can be downloaded as JSON-, Excel- or CSV-file for further evaluation.

Test Result Downloads

Test Projects

Use the Quickstart Wizard to combine a chatbot and test sets into a test project.

Quickstart Wizard

You can start a test session for the test project manually on button click.

Start test session

But in real life projects it is more likely that you will want to integrate Botium Box into your build pipeline. Botium Box provides several integration options:

  • Botium CLI
  • A simple HTTP endpoint for triggering a test project
  • Copy&Paste scripts with cURL or wget

Test results can be delivered as JUnit XML, a format which is supported by all major build pipeline products out there.

Build pipeline integration

Device Labs

Botium Box also supports testing on a user interface level — check that your chatbot is responsive on real devices by combining Botium with a device cloud and browser lab provider like Saucelabs.

Frequently Asked Questions

Please have a look at the Botium website.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store