30 Chatbots: Natural Language Test

Don’t let your chatbot fail at simple tasks

Marina Ashurkina
Jun 29, 2017 · 4 min read

Chatbots are not just running automated scripts, they are made to imitate humans in order to solve tasks more efficiently.

Bots often have a certain personality, that’s why humans unconsciously respond emotionally and expect them to have basic communication skills.

Natural language understanding and chatbots

I tested 30 popular Facebook chatbots to see how well they perform in natural language understanding. There were 5 simple requests: greeting, appraisal, impossible to understand, swearing, goodbye. These are 5 the most basic communicational skills for each bot to understand. Let’s see the results.


‘Hello’ ✅ 18 vs ⛔️12

Greeting is another opportunity to start a conversation

‘Thank you’ ✅ 15 vs ⛔️15

Chatbot User Experience. Randomized replies for ‘Thank you’ (right)

‘You S*ck’ ✅ 10 vs ⛔️20

Here is how different bots handle profanity:

FlightBot handles it with humor 👍
TfL TravelBot is sorry for you thinking so about it

‘Gibberish’

  1. Say that you don’t understand. Period.
  2. Say that you don’t understand and continue to offer some of your services.
  3. Ignore that you don’t understand and offer your services.
You can admit that you don’t understand and offer your primary service again (SkyScanner) or you can ignore the input you don’t understand (Kayak)

‘Bye’ ✅ 14 vs ⛔️16

Different types of goodbye

Conclusion

Check out the table with test results and screenshots below.

Hit ❤️ if you found this article useful.


Results of testing 30 bots

Marina Ashurkina

Written by

Voice Interfaces | now @Huawei | ex cherry.ai | ex api.ai