Botium in a Nutshell, Part 9: Botium Test Levels or Selecting a Botium Connector

Florian Treml
Oct 16, 2019 · 4 min read

This is part 9 of the Botium in a Nutshell series of articles.

Botium supports pretty much every important chatbot technology out there. There are over 20 Botium Connectors available, the number is growing by community and partner contributions.

This article gives an overview what kind of test levels are supported by Botium, and the strengths and weaknesses of each.

Common Chatbot Architecture

This picture shows a typical chatbot architecture:

  • There is a user frontend hosted as chat widget on a website
  • The frontend connects to a backend service (often called Orchestration Service) with web-protocols — HTTP(S), JSON, Websockets
  • Somewhere behind, there is an NLU/NLP engine to convert user input to structured data with intents and entities
  • An additional component is handling the dialogue sessions.
  • An finally, there are some kind of business services, often backed by business databases

Note that in most cases the NLU/NLP engine and the Dialogue engine are combined, for example with IBM Watson Assistant, or Dialogflow. For Microsoft LUIS and Azure Bot Service, those components are spearated though.

The architecture diagrom above shows 6 integration points for Botium test cases.

1: User Interface Level / E2E Testing

Botium supports End-To-End-Testing with a Webdriver Connector, combining the power of Seleniums web browser automation with Botiums chatbot testing superpowers. The chatbot is tested by doing pointing and clicking and typing on a website just as a real user would do.

Scenarios

  • Validating browser/client compatibility
  • Smoke Tests before going live

Pros

  • Testing end user experience
  • Works for all kind of chatbots, independent of the backend technology (in theory)

Cons

  • Requires Selenium infrastructure setup which is not a piece of cake (or use a Selenium cloud provider)
  • Usually, Webdriver scripting is required (in Javascript)
  • Very slow compared to the other testing levels

Part 6 of this series deals with E2E-Testing with Botium

2: API Level / Orchestration Service Testing

Typically, the Orchestation Service is published as an HTTP/JSON or Websocket endpoint. Most chatbot engine providers support such endpoints, and Botium also includes a generic HTTP/JSON connector and a generic Websocket connector adaptable to a wide range of endpoints.

Scenarios

  • Conversation Flow Testing is usually done on API level
  • Integration Testing

Pros

  • Testing on API level reduces flakiness and increases speed
  • Testing near end user experience

Cons

  • No standard API available, has to be adapted for every custom endpoint schema out there

3+4: Backend Level / Dialogue and NLU/NLP Engine Testing

There are lots of dialogue engines out there, some of them even offer free plans. In some cases, those engines include an NLP/NLU engine. Examples for combined dialogue engines are Google Dialogflow, IBM Watson Assistant and Microsoft Bot Framework. Examples for specialized NLU/NLP engines are wit.ai and Microsoft LUIS.

Botium provides connectors for all important dialogue engines out there, as well as for well-known NLP/NLU engines(see Botium Connectors list), and includes a module for providing NLP/NLU analytics.

Scenarios

  • Regression testing for intent/utterance resolution
  • NLP/NLU analytics

Pros

  • Supported out-of-the-box with predefined Botium Connectors
  • Allows very deep analysis of NLP data such as intents and entities

Cons

  • Depending on the implementation, far away from the end-user experience

5: Business Logic Level / Business Service Testing

Usually there is some kind of business service involved, the core to providing value with a chatbot. Botium Box includes an HTTP/JSON endpoint asserter to make sure a test case actually had an impact on business data. For example:

  • By using an eCommerce chatbot, Botium places a test order with the help of an order service in the backend
  • The order service persists the order in the business database
  • Botium asks the order service for details about the order to verify that the order has been persisted

6: Business Data Level / Business Database Testing

This is a special case for the Business Service Testing case above — Botium Box includes asserters for most common business databases (Oracle, PostgreSQL, Microsoft SQL Server, MySQL) to query for test case outcome.

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store