11 f**k ups whilst making a chatbot
We made them so you don’t have to.
We made a chatbot — it lives in Facebook messenger and it’s trying to teach people to cook. It’s called Chop Chop Bot (chat to it here: m.me/chopchopbot or check it out on Facebook fb.me/chopchopbot).
During the build we got lots of things wrong. Some of them were obvious, whilst others came through relentless testing. And although we made a lot of mistakes (11 in total) we learned a lot more in return. Given the relative freshness of chatbots and the rather dull articles out there professing to conclude the future of the technology, we thought there was nothing better than brutal honesty as to the f**k ups we encountered when embracing the Internet’s latest shiny toy:
1. Things can go really bad if you make your chatbot too complicated
In our first version some of the messages were quite long, packed full of information and tasks. We thought this would be useful. But what we found was that no one bothered to properly read the message at all — they just skim read it and just did the last thing you asked. Even grouping together seemingly simple instructions like ‘add some flour and then some oil’ led to people making mistakes or just looking slightly bewildered.
So now Chop Chop Bot breaks down even the simplest task into tiny steps. Rather than just saying ‘peel and chop 2 cloves of garlic’, it will say ‘chop the end off 2 cloves of garlic’’, ‘crush it with your knife’, ‘peel the garlic’ and ‘chop it like this’.
2. Your chatbot is in control of your users brain, so be careful
When people fully tune into a bot, they hand over complete control of their thinking. And common sense can go out of the window, it means your bot needs to cover all of the bases, however small. Following instructions one after another seems to put people into a passive state — and if you don’t tell them to do something, they just don’t do it.
In our initial tests we came across some really serious examples: we needed to remind people to wash their chopping board after having meat on it, and to use an oven glove when getting something out of the oven. So be clear and direct, and don’t forget even the smallest detail.
3. Choose the order of your actions carefully
Don’t say ‘put the bacon in a new frying pan’. If there’s already a frying pan on the stove skim readers will just drop the bacon straight in. Instead say ‘take out a new frying pan and put the bacon in it.’ It’s a subtle difference that creates very different outcomes.
4. Getting the tone wrong can really annoy your users
The alpha version of Chop Chop Bot was called Angry Chef Bot and was modelled, respectfully, on a Scottish celebrity chef we all know. We thought it was hilarious to tell people to chop a f***ing onion, but while some testers found it amusing, in the midst of the cooking experience they wanted someone more helpful by their side. For now our chef bot is chipper and very encouraging. But it’s important that you pick a tone that works best for your bot’s function.
5. Don’t be ambiguous
Chop chop bot checks at various points if you’re on-track by asking if the food looks right or has the correct texture for that point in the cooking process. For one of these checks we simply asked ‘do your fries look crispy and delicious like this?’ In our mind this was equivalent to asking if they had finished cooking. This worked OK most of the time but in one attempt, the test subject’s fries had completely over cooked. And while they were definitely crispy they did not look at all delicious. They replied ‘no’ and Chop Chop Bot told them to put them back in the oven for another 10 minutes. Fail.
6. Don’t forget the visuals
Although we’re using the Facebook Messenger interface, it’s not all about text based communication. If a visual can help tell your story, put it in. When we needed to convey in-depth tasks (such as correctly slicing and chopping an onion) the GIF format worked perfectly. We also found GIFs often worked better than videos as they autoplay in the chat.
7. Your users need constant reassurance
This is particularly important if something isn’t the way it would normally be done. In one instance Chop Chop Bot asks the user to put some bacon in a cold pan while they finish another task — the reason being that we didn’t want to overload the user with too many things to look after at once. This understandably caused a bit of confusion and people thought they’d done something wrong. The solution? Just acknowledge it and be reassuring. ‘Yes we know that’s a bit weird but we don’t want you to have to worry about cooking it while you chop the onions’.
8. Never, ever leave your user alone
If the bot disappears for more than 30 seconds, you’re going to have panicked users. In our case Chop Chop Bot times various parts of the recipe — stirring time for risotto, frying time for onions. We found that even if you tell the user it is going to take 3 minutes, after a minute or so they are starting to panic. So we give them reassurance after the first minute, and the second. People have a tendency to go rogue and cause all sort of knock-on problems if they don’t feel like you are there with them.
9. Find human ways to describe things
There are a lot of legacy things in recipes that may not be helpful in a new format. Is a spoonful always the best way to measure something? In some cases we thought it would be easier (and as long as it wouldn’t f**k up the recipe) so we substituted our instructions with examples such as ‘squeezes of ketchup’ and ‘shakes of tabasco’ whenever possible.
We also called into question the use of time to measure in general. Often the recipe will give a range of time which takes into account the wide variance of cooking equipment, heat, quality etc. But this isn’t very helpful to a beginner. So if a task can be measured in a different way, do that instead. Ask ‘is the bacon crispy?’ or ‘has the rice doubled in size?’ and react and adjust to the response accordingly.
10. If something is critical make it part of the response
At one point in a recipe we swap from meat chopping to salad and the board needs to be washed in between. Our initial approach was to tell the user to wash the board and have a confirmation of ‘done chef’. After having to intervene with one test subject to prevent an E.coli outbreak we changed this confirmation to ‘washed up chef’. We ended up changing a lot of other confirmations to make them more informative and harder to ignore.
11. No matter how hard you try, you can’t plan for everything
We’ve tried to put as much variance control and intelligence into our bot as we could, but there’s no way of predicting everything. Our favourite recipe fail involved one of our brilliant testers accidentally misplacing her spoon…
That’s the most interesting bits of what we’ve learned so far. We’ve a lot more to learn, and we’re excited about where it’s all going.
Please give Chop Chop Bot a go (chat now: http://m.me/chopchopbot, or check it out on Facebook: http://fb.me/chopchopbot) and if you have any feedback we’d be happy to hear it. We’ll be making updates and improvements as we keep figuring out what works and what doesn’t.
Jonny Plackett — SVP of Making Stuff
jonathan.plackett@wk.com
Rob Meldrum — SVP of Experience Strategy
rob.meldrum@wk.com
