Botium in a Nutshell, Part 7: CI/CD Pipeline Integration

Florian Treml
Mar 4, 2019 · 5 min read

This is part 7 of the Botium in a Nutshell series of articles.

This article shows several options how to integrate Botium into your CI/CD pipeline. One of the key principles of Botium is to prevent vendor-lock-in — it is open to the highest degree for integration into any software product out there, even custom pipelines.

Having said that, as long as it supports any of the following technologies, there is a very high likelyhood that Botium will seamlessly fit into your CI/CD pipeline:

  • Running a headless CLI tool
  • Running Mocha/Jest/Jasmine tests with Node.js
  • Calling an HTTP(S)-GET-Endpoint, either by itself or by using tools like cURL or Wget

Botium can output several file formats for further processing in a CI/CD pipeline:

  • JUnit XML (test results)
  • CSV (test results and full test case transcripts)
  • JSON (test results and full test case transcripts)

Prepare BotiumScript test cases

The recommended setup is:

  • Maintain your BotiumScript test cases in a git repository
  • Connect this repository to your CI/CD pipeline (only if you want to trigger the test run on changing test cases)
  • Or connect your chatbot development repository to your CI/CD pipeline — if you are not from the 90ies, this most likely did already happen
  • In the Test step of your CI/CD pipeline, trigger either Botium Box, Botium CLI or Botium Bindings to run the test cases

How this configuration is actually done is very specific to your infrastructure and this article won’t go into detail.

For some guidelines for common CI/CD products see the Frequently Asked Questions section below.

Botium Box

Botium Box provides a flexible HTTP(S) endpoint for triggering tests and delivering the results. After you defined your test project with the Quickstart-Wizard there is a Build Server Integration screen available. It assists in composing the HTTP(S) links and command line calls to use for triggering test case execution.

Build Server Integration screen

Step 1/3: Enter Build Server Integration Scripts Parameters

Provide some information to Botium Box:

  • API Key: API Key to use in your requests
  • Build Id: Build Identifier to name your Test Results (in real-life projects this parameter is dynamically filled from the CI/CD pipeline of your choice)
  • Build Tags: Build Tags to attach to the Test Results
  • Output Format: File format to use for the test results
  • Callback URL: Botium Box will call this URL when test case execution is ready. This is the HTTP(S) endpoint to use for the callback
  • Callback Method: The HTTP(S) Method to use for the callback. If you select POST, then the test case result will be delivered in the request body.
  • Wait with HTTP(S) response until test session completed: When calling the Botium Box endpoint, you can select if you just want to trigger the test case execution and run it in the background, or if the HTTP(S) request should wait until the test session is completed.

The fields in the next section are automatically filled with what you selected here.

Step 2/3: Generated Build Server Integration Scripts

The HTTP(S) Webhook Link is a generic endpoint to use for triggering the test case execution. You can use any client able to do HTTP(S) requests, even your internet browser (yes, you are right: you can start test case execution from just clicking a hyperlink).

  • Start Test Project Link is the link to use
  • Status Link: Use this link to get the status and result for the Botium Test Project (Build Id required)

The other tabs show how to use Botium CLI or the command line tool cURL for doing the same.

For advanced integration, you can build your own HTTP(S) webhook link in the Build Your Own section: The endpoint to use as base is shown there, and the parameters and how to hand them over are described.

All parameters can be handed over to the webhook either:

  • As HTTP query parameter (“…&PARAMNAME=VALUE…”)
  • As JSON body part attribute (Request Body: { …”PARAMNAME”: “VALUE”…})
  • As HTTP header with prefix “X-BOTIUM-” (“X-BOTIUM-PARAMNAME”: “VALUE”)

The parameters are:

  • APIKEY: mandatory — API Key to use in your requests
  • BUILDID: optional at start, mandatory at status check — Build Identifier to name your Test Results, you can choose your own
  • TAG: optional, multiple — Build Tags to attach to the Test Results
  • REPORTER: optional — File format to use for the test results (json, junit, csv)
  • CALLBACKURL: optional — HTTP(S) endpoint to use for the callback
  • CALLBACKMETHOD: optional — HTTP(S) Method to use for the callback
  • WAIT: optional — 1 (wait until complete) or 0 (run in background)

If everything is setup correctly, the test case results will be imported by your CI/CD pipeline. Nevertheless, you can always inspect your test results in detail in Botium Box.

Test Case Result with Transcripts

Using Botium CLI

With regards to CI/CD pipeline integration, there is nothing special with running Botium CLI. You should add a botium.json file to your test case repository and add a script task to your CI/CD pipeline:

> botium-cli run --config path/to/botium.json --convos path/to/files

Botium CLI can output several file formats, use the command line help for more information.

Using Botium Bindings

Most CI/CD pipelines support the well-known test automation frameworks Mocha, Jest or Jasmine. Consult the documentation of your product on how to run those test cases in your test case repository built with Botium Bindings.

Frequently Asked Questions

Does it support Bamboo ?

Yes. Bamboo has excellent support for scripting, so best option for running test cases on Botium Box is to use a script task with cURL or Wget.

And remember to map the build environment variables to the Botium Endpoint parameters — you definitely want to map the bamboo.buildKey variable to the Botium Box BUILDID parameter!

Does it support Jenkins ?

Yes. For running test case on Botium Box, either use a simple shell script with cURL or Wget, or use an HTTP Request Plugin. Botium output as JUnit XML can be visualized with the Test Results Analyzer Plugin.

See here for instructions.

Does it support Codeship ?

Yes. Just use a script task with cURL or Wget.

Does it support Travis ?

Yes. Just use a script task with cURL or Wget.

Does it support build pipeline product X from vendor Y ?

Most likely, yes.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store