Let’s Chat About Testing Chatbots

My experience testing botbuilder bots and building the Bot Tester framework

Testing bots using Microsoft’s Bot Builder Framework has been one of the more painful parts of using their framework. For quick testing, the easiest method has always been spinning up a bot and testing in the emulator. Message goes in, message comes out.

Unfortunately this approach has several problems:

• It is slow
• It is not possible to automate and scale
• It gives no insight into session state

I assumed that someone else had experienced this before and had a programmatic way to test their bots. After a surprising amount of searching, the most relevant solutions I found referenced the botbuilder dialog’s tests:

An abridged version of the Dialog tests inside the botbuilder source

While this seemed like the best solution I could dig up, it was difficult to follow, required use of the console connector to send messages through it, required knowledge of the current step in a dialog waterfall, and (assuming the dialog itself is not modified as part of the test) could not inspect the session at any point.

After much frustration trying to finagle this switch based approach to test what I needed, I decided to make my own testing framework. My goal was to have a framework be reliable to the point that I would not need to open an emulator once to be sure of my bot’s behavior. To do that, I set the following requirements:

• It must accept any bot, regardless of connector
• It must be able to send a string or IMessage to the bot and run expectations on the response(s) as IMessages
• It must be able to inspect session state whenever the bot is in an idle state

After a few weeks of hacking, tweaking, and receiving helpful feedback, the framework now meets all the requirements and some more! Here are some examples:

• It can send strings to the bot and run expectations against the response

• It can inspect session state

• It can send IMessages to the bot and run expectations against the response

• It has the familiar Promise .then syntax to perform work between steps

• It can run expectations against regular expressions

Check out the source at github.com/microsoftly/BotTester if you’re interested in seeing how it works or looking at the the in depth docs!