30 Chatbots: Natural Language Test

Don’t let your chatbot fail at simple tasks

Marina Ashurkina
4 min readJun 29, 2017

--

Chatbots are not just running automated scripts, they are made to imitate humans in order to solve tasks more efficiently.

Bots often have a certain personality, that’s why humans unconsciously respond emotionally and expect them to have basic communication skills.

Natural language understanding and chatbots

It’s up to developers and designers. They have a choice to support natural language understanding or not. If they decide to support it, there are plenty of free tools to do so. Some of them are wit.ai, api.ai, IBM Watson, Microsoft Bot Framework.

I tested 30 popular Facebook chatbots to see how well they perform in natural language understanding. There were 5 simple requests: greeting, appraisal, impossible to understand, swearing, goodbye. These are 5 the most basic communicational skills for each bot to understand. Let’s see the results.

‘Hello’ ✅ 18 vs ⛔️12

Hello is one of the most popular user requests to bots. From 30 tested chatbots only 18 responded to ‘Hello’ correctly, others didn’t understand it.

Greeting is another opportunity to start a conversation

‘Thank you’ ✅ 15 vs ⛔️15

Sadly, 15 bots couldn’t respond appropriately to the appraisal of their own work. The other 15 replied correctly. Some of them used the same reply over and over, some of them had a lot of interesting replies and sent them randomly.

Chatbot User Experience. Randomized replies for ‘Thank you’ (right)

‘You S*ck’ ✅ 10 vs ⛔️20

I know it’s not very nice to say ‘You s*ck’ to a bot and I didn’t mean it. But developers can’t ignore user’s swearing. This happens very very very often.

Here is how different bots handle profanity:

FlightBot handles it with humor 👍
TfL TravelBot is sorry for you thinking so about it

‘Gibberish’

Bots will always fail to understand some percentage of users’ input. This is normal. There is no right or wrong way to handle such requests but here are 3 different approaches companies usually use:

  1. Say that you don’t understand. Period.
  2. Say that you don’t understand and continue to offer some of your services.
  3. Ignore that you don’t understand and offer your services.
You can admit that you don’t understand and offer your primary service again (SkyScanner) or you can ignore the input you don’t understand (Kayak)

‘Bye’ ✅ 14 vs ⛔️16

Saying goodbye is as important as saying Hello. If the user had a pleasant experience with your bot, they will say goodbye. Is your bot prepared? Well, 16 out of 30 bots I tested were not prepared to tell me goodbye and treated it as unrecognized input. Even though they allow natural language input.

Different types of goodbye

Conclusion

Natural language understanding is essential for a chatbot. Your bot won’t be able to understand everything at once, learning requires time. But even if you release your first version of a chatbot, do not forget to include these 5 crucial communication skills.

Check out the table with test results and screenshots below.

Hit ❤️ if you found this article useful.

Results of testing 30 bots

--

--

Marina Ashurkina

Follow for Conversational AI Insights | Chatbots | AI Agents | AI Copilots | Voice Assistants. Building Chatbots since 2012