This is part 5 of the Botium in a Nutshell series of articles.
- Part 1: Overview
- Part 2: Writing Chatbot Tests
- Part 3: Automating Chatbot Tests
- Part 4: Advanced Topics
- Part 5: The Botium Box
- Part 6: E2E-Testing with Botium Box
- Part 7: CI/CD-Pipeline Integration
- Part 8: Best Practice for Test Case Development
- Part 9: Botium Test Levels or Selecting a Botium Connector
- Appendix A: Botium Wiki
- Appendix B: Botium Connectors
This article is an introduction for Botium Box users. There is a trial and a free community edition available of the Botium Box. The best way to explore Botium Box is to start with the trial, please register here.
Botium Box Taxonomy
As a starter a short overview of the terms used in this article an in the Botium Box:
- Test Cases are written as Convos and Utterances — see here
- A Test Set is a collection of Test Cases
- A chatbot in Botium Box is a set of capabilities to tell Botium how to connect to a single instance of a chatbot
- A test project is a combination of test sets and a chatbot — the specification of a test environment
- A test session is the process of sending test sets to a chatbot and asserting the correctness of the results
- A device set is a collection of device or browser specifications to be used for user interface testing
- An agent is a piece of software for test execution. The more agents are connected to the Botium Box, the more tests can run in parallel.
The Dashboard is the entry point. You can see an overview how the quality of your chatbot evolves over time, how the chatbot performs in your environments and what test sets perform well.
Connecting your chatbot
Botium supports a large variety of chatbot technologies. For each technology, the configuration form contains inline help how to retrieve the required connection information, tokens, secrets etc.
You can always switch to the Advanced Settings mode to get fine tune the capabilities.
For checking your configuration, you can always start a live chat with your chatbot.
Just click CONNECT to start Botium Core in the background and talk to your chatbot.
You can save the conversation as test case. This is the quick way to start buildling your custom test set.
The Test Set Library and Humanification
Test case files are managed in test set libraries. Within a single test set, you can combine test cases from various sources:
- Write Convos and Utterances in Botium Box
- Upload Excel files
- Register a shared folder holding the test cases
- Register a Git repository holding the test cases
Note: in real-life scenarios, the last option (Git repository) is clearly the best option as it allows the test cases to be unter version control.
When cloning a test set, you can optionally apply various humanification algorithms: the test set is then modified to take human habbits into account and to appear more like human input to the chatbot:
- space and newline handling
- sausage fingers
- common typographic errors for qwertz/qwerty keyboards
- case sensitivity
- and many more …
Starting a Test Session
On the Test Results screen, a test session can be started. Depending on the number of test agents, the performance of the chatbot and the number of testcases this will take some time.
Exploring the Test Results
The test report shows on a very detailed level every aspect of the test execution:
- Successful and failed test cases
- Transcripts for all executed test cases
- Detailed log files for all test execution jobs
- Performance measurement
The test results can be downloaded as JSON-, Excel- or CSV-file for further evaluation.
Use the Quickstart Wizard to combine a chatbot and test sets into a test project.
You can start a test session for the test project manually on button click.
But in real life projects it is more likely that you will want to integrate Botium Box into your build pipeline. Botium Box provides several integration options:
- Botium CLI
- A simple HTTP endpoint for triggering a test project
- Copy&Paste scripts with cURL or wget
Test results can be delivered as JUnit XML, a format which is supported by all major build pipeline products out there.
Botium Box also supports testing on a user interface level — check that your chatbot is responsive on real devices by combining Botium with a device cloud and browser lab provider like Saucelabs.
Frequently Asked Questions
Please have a look at the Botium website.
Looking for contributors
Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here: