6 easy ways to break a chatbot
Or searching for the “chat” in chatbots
It’s an understatement to say that chatbots are popular. New chatbots are created and deployed every day and multiple papers and posts are published on that topic. Superlatives are commonplace, and some claims are a bit outrageous, like “chatbots will replace IVR”, “chatbots are the new apps”, and even “chatbots are the new black”.
Our team has been working on a text chatbot effort for a few months now, and we quickly saw how complex it can be to create a good conversational chatbot. Although we have a good idea of what we ultimately want to accomplish and how to get there, as a UI designer, I’ve been looking at other text chatbots, mostly consumer bots, to get inspired and see what’s out there. So far, this has been an underwhelming endeavour. For starters, most chatbots pretending to be conversational are not; buttons are not conversational. A conversational interface must involve the possibility for the user to type their response, freely, most of the time (unless it’s more efficient and logical to use buttons), and the chatbot should handle that response in a sensible manner. Sounds reasonable, right?
When trying chatbots, I found myself using a few basic tests to determine whether the chatbot was worth investigating further. I’m not talking about anything close to a Turing test (a recent post by Kumar Shridhar explores that topic and concludes that there is a long way to go before achieving that), just a few seemingly easy things. Many chatbots did not make the initial cut.
Disclaimer: this is not scientific at all and I’m not claiming that the observations below are statistically significant.
Here are 6 quick tests that unfortunately often fail:
Typing what is displayed on the button
One of the very first things I try is to answer the question by typing the exact same text as it appears on one of the displayed buttons. In several cases, that’s enough to get an error message from the chatbot. One particular chatbot doesn’t recognize any displayed answer when it’s typed instead of using the buttons, and it nevertheless introduces itself as conversational.
Asking for help
In many cases, it’s not entirely clear what the user is supposed to do or type. So I type things like “what can I do” or “help” to try and get some tips. This is not handled by a good proportion of existing chatbots.
Answering the question
Many chatbots make suggestions or propose to send daily notifications. For example, a recipe chatbot, after providing sample recipes for an ingredient that I entered, asked me “Want a tidbit like this everyday?”. I just answered “nope”, and it came back with “Sorry, I couldn’t find any nope recipes”. This was a yes/no question, so I think it’s perfectly reasonable to expect a variant of yes/no to be understood. Lately, they “improved” it: I typed “no” and got this interesting response: “I’ve got you covered! Here are several delicious no recipes.” These included spare ribs and whipped cream…
Answering outside the buttons
Most chatbots provide choices in the form of buttons. However, it often seems pretty reasonable to answer something related to the question, but not necessarily displayed. For example, a chatbot where one can order a beauty product basket asks for whom the gift is. Among the choices are mother, sister, friend, and a few others, including “not a woman”. I answered “my husband”, thinking that they should have some men products as well, it’s a big cosmetics company. The chatbot not only did not accept “my husband” as a valid response, it did not even recognize the word. Anticipating possible answers and reacting reasonably should be part of any conversational design.
Starting over
Sometimes we reach a dead end and want to start over. In that case, typing things like “reset” or “start over” seems quite intuitive. Unfortunately, that doesn’t always work and is often treated as an invalid input.
Saying goodbye
While a majority of chatbots handle this request well, it’s not always the case, and I think it’s so basic that it should probably be the very first thing to include when designing a chatbot.
I can’t claim to be an authority in chatbots but I certainly know my way around conversational user interfaces, and many of the chatbots I’ve seen don’t even come close, which is unfortunate because high expectations have been set and are not met. I consider that conversational chatbots should be approached in a similar manner as voice interfaces and should comply with basic user experience criteria. They should also be tested with real end users, in usability tests. But that’s another topic.
