Coding a chatbot builder platform, Part 1: How to make a dumb chatbot
How I approached the challenge of programmatically creating “conversations”.
Last week I decided to build a bot builder application. The goal was (and still is, as the project will be ready next week) to make an app that serves as a platform for users of all levels to build simple chatbots and host them on my site. In this two-part blog post I’ll cover my process and some valuable sources for building a platform for bots — first stupid ones, then, feature by feature, smarter ones. (When I write stupid, I don’t mean lame or bad. My chatbots are cool, but they’re far from intelligent)
What makes a minimum viable chatbot?
In all simplicity, finding the right response to a message is going through as many as possible checks before eventually offering the fallback answer: “I don’t understand”. I decided to take this approach to build my chatbot. That way I can keep on adding more layers of “does the message match this piece of the stuff my bot knows” to my minimum adequate chatbot. Here were my initial goals:
- Users can script their own dialogues. A user can add “triggers”, meaning words or sentences, that their bot will respond to. They’ll be able to specify an array of different responses, as many as they like. If the bot is able to match a received message to it’s triggers, it will choose a random response. This forms the “scripts” of each bot.
- Bots will be able to offer general scripts (such as greetings and answers to common questions) and fallback “I don’t understand” responses if the user chooses to include them
- Bots should be decent at understanding what the user is telling them, even if it’s not an exact match to known triggers.
These characteristics make my bots “retrieval-based”, as this excellent article describes. Retrieval-based conversational models are easier to implement, as they use a repository of predefined responses and some rules to choose them. The other option would have been to build chatbots that are able to generate their own responses, but that approach would have been much more time-consuming and prone to grammatical errors. It would have been unclear to me how to offer users the chance to script these bots with their custom input.
Building the bot builder
The simple goals translated well to actual incremental technical features. The scripting platform, which I implemented with React, is basically a dynamic form that grows and shrinks by user’s choosing. There’s no limit to how many triggers or responses you can add. At scripting phase, users can test out their bots and then save them once they’re pleased with their scripts. They’ll be able to edit the bot, chat with it, or share a link to anyone to chat with the bot, as it’s hosted on my site. Every message the bot receives, is sent to my Ruby on Rails backend, where a matching process tries to find the right response and sends it to the front-end (user).
At this stage, I had already implemented some simple string “cleaning”, like removing non-alphabetic characters and whitespace, but the bots were miserable at understanding other than very exact matches. They said to “I don’t understand” a lot, and this wasn’t a great user experience. It really highlighted the fact how difficult it is to mimic even simple human dialogue programmatically.
After choosing the base logic that all my bots should implement, I set to scripting default triggers and responses. On the frontend bot-scripting platform, users could opt-in on small default scripts, such as greetings, goodbyes, easy questions (e.g. “how are you?”) and existential questions ( e.g. “what are you?”). The bots got slightly less stupid after this simple addition.
Fuzzy string matching for the win
The final step for my dumb bot maker was to implement “fuzzy string matching”. There’s an amazing ruby gem and library for that. Instead of comparing whether string1 == string2, fuzzy string match calculates the distance between two strings. The gem uses the the Jaro-Winkler distance algorithm to return a value between 0 and 1 to represent similarity between the strings. 1 means the words are the same, 0 means they’re not similar at all. I set my bots’ default match threshold to 0.8, because that seemed to get past typos or extra filler words while not fully missing meaningful differences. Initialising a comparison instance was easy (fuzzy_match = FuzzyStringMatch::JaroWinkler.create(:native) ) and so was using it.
My final “find the right response to this message”-method includes all of these three checks. I made some rules to prioritise the responses, in case some user message would match more than one trigger. At this point, these rules simply manifest in the order of my matching method.
1) matches to user’s scripts
2) default script matches
3) find clues from previous conversation (not yet implemented)
The three basic features together took the bots to a quite adequate level. I was happy to see my bots weren’t too bothered by typos, knew to prioritise user’s scripts, and knew to offer default answers to some common messages. I have experimented with basic and beginner friendly machine learning implementations to add extra understanding to my bot builder, which I will document for the next part of this blog post.
