Botium in a Nutshell, Part 6: E2E-Testing with Botium Box

Florian Treml
Feb 12, 2019 · 8 min read

This is part 6 of the Botium in a Nutshell series of articles.

This article shows how to set up Botium Box as an End-to-end-testing environment connected to a device lab or browser lab. For an introduction to Botium Box, please see Part 5 of this blog series.

Your chatbot needs testing, you know that — otherwise you wouldn’t read this article. Apart from the conversation flow, which is best tested on API level (see Appendices for a list of supported APIs and SDKs), you want to make sure that your chatbot published on a company website works on most used end user devices out there. Botium Box helps by connecting your BotiumScript test cases with your chatbot widget using a Selenium server, the current industry standard for testing web applications.

Step 1: Connect to device lab or browser lab

Botium Box supports cloud providers, but you can connect to your self-hosted Selenium server as well.

  • cross browser testing cloud
  • real device cloud
  • Experitest Mobile App & Cross-Browser Testing End-to-End
  • Local, self-hosted Selenium server
  • Integrated PhantomJS (virtual browser)
Device Provider Settings

The Selenium Endpoint has to be filled with URL of the Selenium server. For cloud providers, consult the documentation or contact the cloud provider support. You will have to enter username and password or access key as well. For local installations, consult your developers.

  • For Saucelabs, the Selenium Endpoint typically is, the Username and the Access Key you can find on the User Settings section in the Saucelabs dashboard
  • For Saucelabs Real Device Cloud (former TestObjects), you have to create a Test App first, switch to the Appium view and go to the Setup Instructions — you will find the Selenium Endpoint and the API Key there
  • For Experitest, you can find the Selenium Endpoint (typically and the Access Key in the Seetest Dashboard in the Automation view

See here for a guide how to setup your own Selenium Grid and connect it to Botium Box.

Step 2: Compose Device Sets

Next Step is to select the devices for running your test cases. Botium Box comes preconfigured with reasonable default device sets. There are several guidelines available what devices you should select for your use case.

Device Sets

For some cloud providers, the available devices and browsers are shown for selection.

Device Selection
  • For Saucelabs and TestObjects, the device list is retrieved in real-time from the Saucelabs API
  • For Experitest, the file EXPERITEST.json in the resources folder is used to show the selection of the available devices and browers
  • For a local selenium grid, the file LOCALSELENIUM.json in the resources folder is used to show the selection of available devices and browsers

See here for a guide how to add the available browser configurations for your local Selenium Grid to Botium Box.

Step 3: Prepare Botium Webdriver Connector

To finish the configuration process, you now have to tell Botium Box how to connect to your chatbot. Make sure to select WebdriverIO as chatbot technology.

Chatbot Connection Settings

If you ever worked with Selenium, you are aware that writing an automation script usually is a time-consuming task. Botium Box speeds up the development process by making reasonable assumptions:

  • There maybe is some kind of click-through to actually open the chatbot
  • The chatbot has an input text field, or some buttons to click (however it looks)
  • The chatbot outputs reponses in some kind of list within a window (however it looks)

All those assumptions can be parameterized for adapting it to your actual chatbot website.

Webdriver Connector Settings

You can instruct Botium Box to take a screenshot of every test case on each device to get an overview of the visual appearance of the chatbot widget when running your test cases.

Configuration for Common Chatbot Widgets

For some common chatbot widgets and websites, Botium provides out-of-the-box Selenium scripts (Microsoft Bot Framework Webchat, IBM Watson Assistant, and more to come …) — Botium will be able to detect the widget on the website automatically, based on Selenium selectors. Just select the technology in the Webdriver Script field.

Custom Configuration with Selenium selectors

With some know-how with Selenium Botium can be adapted to almost every webchat out there. The first thing to do is to analyze your chatbot widget to find out about the Selenium selectors to use — here is a guide what to look out for. The common Selenium flow in Botium is:

For rather simple chatbot widgets, the input and output selectors can be entered in the respective fields in the Botium configuration screen. If there are dynamic actions required to open the chatbot window (Point 2 from above), you have to do some Javascript coding and register the scripts in Botium Box.

Custom Configuration with Javascript Snippets

There are several extension points where you can inject your own Javascript snippets. In most cases, the extension point you will have to adapt is the WEBDRIVERIO_OPENBOT extension point: it is responsible to open the chatbot widget and make it ready for Botium to start the conversation.

Here is an example:

  • it waits for a cookie consent message to appear
  • clicks it away
  • then waits for a chatbot button to appear
  • clicks it
  • and waits until the chatbot window and the element on it are actually visible.
module.exports = (container, browser) => {
return browser.$('.cc-btn')
.then(ccBtn => ccBtn.waitForDisplayed(20000).then(() =>
.then(() => browser.pause(2000))
.then(() => browser.$('#StartChat'))
.then(startChat => startChat.waitForDisplayed(20000).then(() =>
.then(() => browser.$('#chat').then(c => c.waitForDisplayed(10000)))
.then(() => browser.$('#textInput').then(t => t.waitForDisplayed(10000)))
.then(() => browser.$('.from-watson').then(f => f.waitForDisplayed(10000)))

You can find more samples in the Github repository for the Botium connector.

Adding your Javascript Snippets to Botium Box

Please the snippets you create in a file (openbot.js) and place it in the resources folder of the Botium Box distribution (or a subfolder — organize it however you want).

Then, in the chatbot view, switch the Connector/Chatbot Technology to Other (Advanced Settings).

The configuration UI is now switched to a plain view of the Botium capabilities. Use the Register new capability button at the bottom to add a new capability row.

  • As capability name, enter the name of the extension point (WEBDRIVERIO_OPENBOT, …).
  • As type, select Text
  • As value, enter the relative path to the Javascript snippet (in our example from above: resources/openbot.js)

Click on Save to finish the Selenium configuration for your chatbot.

More Advanced: Add Javascript Code as Capability

You could add the Javascript code itself as capability value by setting the value to the Javascript code:

result = browser.$('.cc-btn')
.then(ccBtn => ccBtn.waitForDisplayed(20000).then(() =>
.then(() => browser.pause(2000))
.then(() => browser.$('#StartChat'))
.then(startChat => startChat.waitForDisplayed(20000).then(() =>
.then(() => browser.$('#chat').then(c => c.waitForDisplayed(10000)))
.then(() => browser.$('#textInput').then(t => t.waitForDisplayed(10000)))
.then(() => browser.$('.from-watson').then(f => f.waitForDisplayed(10000)))

Most important to know is to assign the result of the Javascript code to the global result variable.

Step 4: Run Test Cases

When running test cases, you have to select the device set(s) to use.

Device Set(s) selection

Apart from that, it behaves as any other test project in Botium Box. Click on Save and Start Testing to start the first test run.

Step 5: View Test Report

The generated test report shows every single detail of the test run, including the transcript, the expected outcome and, probably one of the most interesting information items, the screenshot(s) taken by Botium Box.

Botium Box Test Report

Frequently Asked Questions

Why should I test my chatbot user interface ? Is it not enough to test the conversation flow on API level ?

You should at least make sure to verify the visual appearance and the correct operation of your chatbot user interface on most used desktop and mobile device browsers out there. Botium Box can support you in this.

If you already have Selenium scripts for testing your website including the chatbot widget on different browsers, then this already seems to be a good setup, and it most likely won’t make sense to duplicate the tests in Botium Box.

What is the difference to just using Selenium ?

Selenium is just an API connecting custom automation code to desktop or mobile device browsers. It is the technology backing the Botium Box Webdriver connector. Botium Box adds device management, test case management, test reporting, load balancing and much more to plain Selenium.

Do I need developer skills to do E2E-testing with Botium ?

It depends. If you are using one of the well-known and widely used webchat widgets out there (for example, the Bot Framework Webchat), then the answer most likely is no, you don’t need developer skills. For custom webchat widgets, you need at least the skills to define Webdriver selectors.

I have my own Selenium server, can I use it with Botium Box ?

Sure, see Botium Wiki.

I have an account for mobile device cloud XYZ, can I use it with Botium Box ?

If the device cloud provides a Selenium endpoint, then most likely the answer is yes. Please contact us to clarify.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store