Autobot: Effective End-to-End Testing of Bots
The tooling for bots has come a long way. Today, with Luis, API.ai, and Wit, and Bot Framework — building a bot is much better experience than it has ever been before. However, we felt there was one critical component that was missing: end-to-end testing.
Bots are amazing products because of their flexibility to respond to wide ranging and ambiguous interactions. However, In the world of natural language, almost everything is an edge case, making effective testing key to delivering a good service.
To address this problem, we at Aiden.ai created Autobot, a multi-platform, extensible, automated bot testing framework. With Autobot, you write a dialogue in plain english. Autobot will then connect to your bot and play this dialogue with your conversational AI, asserting that all the responses match the script.
We use Autobot to automatically run our bot Aiden through hundreds of different simulated scenarios before each release. In line with our commitment to the open source and bot community, we are open sourcing Autobot under the MIT License. Fork us on github.
Below is a very simple examples of our plain english dialogue file format. It is human/readable and writable.
But it comes with many advanced features such as variables, wildcards and branching:
How it works
Autobot searches for simple dialogue files, which are plain text scripts, written in the human readable and writable YAML format. It then emulates a fake user, and acts out the scripted dialogue with your bot. It sends out messages and listens for how the bot replies.
Some of the core features include:
- The ability to interact with any bot platform. Just implement two functions
sendandonReply. - It can run hundreds of tests in parallel.
- The ability to support matching complex patterns in the dialogue, with full regular expression support, and also match against a set of valid responses.
- Load parts of the dialogue from an external file to prevent you writing all your dialogue twice.
- Test complex branching conversations where the human or bot agent could go down multiple paths.
Handling Branches
One of the most interesting challenges we had to solve was that complex bots have complex behavior. Just like a person, when you ask it a question, sometimes you aren’t sure how it will respond. The type of response can completely change the direction of the conversation.
For example, if you ask Aiden ‘How many countries is this campaign live in?’. One reply might be ‘Your ad campaign is live in 10 countries right now’. A user might then drill down, and ask more questions like, ‘Which countries?’ or ‘Show me ad spend by country’. However, Aiden might also reply to the original question ‘Your ad campaign is not live anywhere right now’, which is a correct response but, to which the drill down questions do not make sense.
Additionally, bots can ask questions too, like “I’ve detected that your ad spend is 20% higher than last week, would you like to know more?” We need to test both the “Yes” and “No” responses, but don’t want to write two separate test files just for this choice.
To solve this, we model each dialogue file as a directed acyclic graph, where every node is a complex regex. We then walk the tree to figure out what will be the maximum number of users required to walk all the paths are. Each virtual user will try follow a unique path through the conversation. This lets you test complex behaviors without requiring repetitive dialogue files with large copy & pasted sections.
Try it out
So check us out on Github today: https://github.com/aiden/autobot
It ships with a connector to the Microsoft Botframework. But implementing a connection to your own bot is only a matter of filling out the send and onReply functions. If you need any help, just drop us an issue on Github. Also, we’d be happy to accept pull requests if you have ideas on how to improve autobot for everyone!
At Aiden.ai, we are building a virtual colleague for marketers. We’re excited to contribute back to the open source community, and help everyone build better bots. Please don’t hesitate to show your support by clicking the 👏 button below!
