30 Chatbots: Natural Language Test
Don’t let your chatbot fail at simple tasks
Chatbots are not just running automated scripts, they are made to imitate humans in order to solve tasks more efficiently.
Bots often have a certain personality, that’s why humans unconsciously respond emotionally and expect them to have basic communication skills.
Natural language understanding and chatbots
It’s up to developers and designers. They have a choice to support natural language understanding or not. If they decide to support it, there are plenty of free tools to do so. Some of them are wit.ai, api.ai, IBM Watson, Microsoft Bot Framework.
I tested 30 popular Facebook chatbots to see how well they perform in natural language understanding. There were 5 simple requests: greeting, appraisal, impossible to understand, swearing, goodbye. These are 5 the most basic communicational skills for each bot to understand. Let’s see the results.
‘Hello’ ✅ 18 vs ⛔️12
Hello is one of the most popular user requests to bots. From 30 tested chatbots only 18 responded to ‘Hello’ correctly, others didn’t understand it.
‘Thank you’ ✅ 15 vs ⛔️15
Sadly, 15 bots couldn’t respond appropriately to the appraisal of their own work. The other 15 replied correctly. Some of them used the same reply over and over, some of them had a lot of interesting replies and sent them randomly.
‘You S*ck’ ✅ 10 vs ⛔️20
I know it’s not very nice to say ‘You s*ck’ to a bot and I didn’t mean it. But developers can’t ignore user’s swearing. This happens very very very often.
Here is how different bots handle profanity:
‘Gibberish’
Bots will always fail to understand some percentage of users’ input. This is normal. There is no right or wrong way to handle such requests but here are 3 different approaches companies usually use:
- Say that you don’t understand. Period.
- Say that you don’t understand and continue to offer some of your services.
- Ignore that you don’t understand and offer your services.
‘Bye’ ✅ 14 vs ⛔️16
Saying goodbye is as important as saying Hello. If the user had a pleasant experience with your bot, they will say goodbye. Is your bot prepared? Well, 16 out of 30 bots I tested were not prepared to tell me goodbye and treated it as unrecognized input. Even though they allow natural language input.
Conclusion
Natural language understanding is essential for a chatbot. Your bot won’t be able to understand everything at once, learning requires time. But even if you release your first version of a chatbot, do not forget to include these 5 crucial communication skills.
Check out the table with test results and screenshots below.
Hit ❤️ if you found this article useful.
