Regression Testing an NLP model with Microsoft LUIS and Botium

Florian Treml
Feb 4, 2019 · 3 min read

In the age of agile development, the age of manual regression testing is over. We developed a Botium extension to enable an agile developer team to run automated regression tests on a Microsoft LUIS model within their CI Pipeline.

2019/01/20 Chatbots are driving the industry. With Botium we are driving chatbots. After months of hard work we are finally there: we are proud to announce the general availability of the newest Botium Stack member, the Botium Box. Check out some game changing features for testing and training of chatbots with our Community Edition!

Why ?

When doing continuous improvement of an NLP model — in our case, for a chatbot — you have to make sure to not mess the existing model up. Especially when dealing with similar user utterances mapping to closely related intents there is high risk involved on training additional user utterances. Same applies for utterances with closely related entities.

What we need is to make sure that our most important intents and entities are recognized for the most important user utterances. We need regression testing. We just published a Botium connector for doing this with Microsoft LUIS. For an introduction to Botium, the Selenium for Chatbots, please read the “Botium in a Nutshell” blog series:

The rest of this article describes the steps required to setup a project for holding the test cases and test runners for your Microsoft LUIS model.

Before you begin, make sure you have a current version of Node.js installed on your workstation.

Botium Installation and Initialization

Botium Box comes with out-of-the-box support for Microsoft LUIS, so you don’t have to manually setup those components.

Run these commands to install Botium and Mocha, to create a Botium project skeleton and run the skeleton project.

> npm install botium-bindings
> npm install botium-connector-luis
> ./node_modules/.bin/botium-bindings init mocha
> npm install && npm run mocha

Follow these steps to change the file botium.json to connect your Microsoft LUIS model to Botium. Create a file lights_on.convo.txt in the spec/convo folder:

lights on

This BotiumScript asks your Microsoft LUIS model for resolving the user utterance “set lights concentrate” and makes sure that it resolves to the expected intent and entities.

With BotiumScript, you don’t have to write files for all utterances, you can use utterances files to collect all utterances to resolve to the same intent and entity into a single file.

Running Test Cases with Mocha

Running the test case is as simple as:

> npm run mocha

Either trigger the tests manually, or setup your CI Pipeline to run them — it’s up to you now.

Running Test Cases with Botium Box

The Botium Box contains a nice configuration wizard for Microsoft LUIS connector. You can integrate Botium Box into your CI Pipeline as well.

Botium Box Dashboard

Please see this article for an introduction to Botium Box:

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store