In defence of the chatbot

Remember chatbots? That thing everyone talked about before blockchain swooped in and stole the show? Well, according to Wired, Chatbots are dead. Brands already cooled on chatbots in 2017 and the shutdown of M, a Facebook Messenger bot which automatically completes tasks for users, might be the final nail in the coffin.

But should we really be dismissing chatbots that quickly?

We caught up with John Keefe, bot developer at Quartz; Eduardo Suárez and Miguel Eduardo Gil Biraud from Politibot, a chatbot platform that was created in 2016 for the Spanish elections; and Philipp Naderer from Austrian public broadcaster ORF who created the Wahl-bot for the Austrian presidential elections — all to get a better idea of where chatbots stand in early 2018, whether newsrooms should invest in them, and how to make a chatbot live up to our expectations.

Hello, chatbot. Why are you so clueless?

‘We would all love to live in a world where having a conversation in natural language with a machine is possible and works flawlessly. We have come a long way and the field is moving very quickly’, said Suárez from Politibot. ‘But unfortunately we are not there yet. It will be years until it works well enough to not cause rejection among the general population’.

And rejection it does cause. After Facebook unveiled its API in 2016, news organisations and brands were able to have their own chatbots up and running in a matter of minutes. Unfortunately, these chatbots are reported to have a failure rate of 70 percent as they don’t work without human intervention, and their user experience is often clunky and awkward.

Source: https://chatbot.fail

Worse still, a chatbot can turn into a chatmonster within minutes. Microsoft’s AI chatbot Tay, whose aim was to interact with millenials to improve its conversational skills through machine learning, quickly learned to parrot holocaust denying, racist, and anti-female rhetoric.

‘If you use actual human data to train a robot, it’s inevitable that it will pick up the habits of humans, including the bad ones’, wrote Chris Mills, news editor at BGR, in a piece about why all Microsoft chatbots are ‘assholes’. But he also pointed out that if you were to remove certain data points, the AI will be less likely to understand human conversation and behaviour down the line, suggesting that a fully functioning and prejudice-free AI chatbot is a tall order. (This op-ed in the Guardian about machines and prejudice provides for a good read.)

Many news chatbots aren’t actually all that chatty either. Try the CNN chatbot and you’ll be given a few buttons to click on, which all trigger an automated answer or article suggestions, creating a safe but limited user experience. A number of news chatbots, including Australia’s ABC News, mostly send out newsletter-style digests every day rather than engaging in a human-like messenger-style conversation, which could be made possible through natural language processing (NLP).

The team from Politibot points out that vocabulary related to news is vast and varies greatly depending on what is newsy at any given time. Suárez and Biraud told us that it’d be easier to add NLP to a washing machine customer support chatbot than to a news chatbot, because the former would only require limited vocabulary related to washing machine parts (and maybe a couple of responses to insults when a customer’s house is flooded because of a broken washing machine).

‘Even if we could make a news bot work well, and that’s a big if, the reality is that language models exist mostly in English and if you were to train your own models they would need massive amounts of data’, said Suárez and Biraud in a joint email response. ‘If we’re talking about other languages, such as Spanish, there are no good models yet, but they may be coming in the next few years. The story is even more dire for languages like Estonian or Icelandic where the amount of usable data to train models is scarce’.

This is why it seems like most chatbots are currently command based with sprinklings of NLP.

What then makes a good chatbot?

1. Strike the right balance

‘Useful and enjoyable chatbots have to be good at solving their core tasks, but they need to extend into a creative machine — if the user wants them to do so. Finding the right balance between commands and open conversations is a central part of every chatbot design process’, said Naderer.

ORF’s chatbot Wahl-bot, a newsroom experiment used during the Austrian presidential election, proved popular, receiving 85,000 messages on election day alone. The bot was designed in a way that kept the user within the set of commands supplied by the chatbot, mostly in the form of buttons. But Naderer tells us that they also came across more curious users who wanted to explore the chatbot in different ways: What else can I ask the Wahl-bot? What are its limits?

Screenshot from Wahl-bot. The user can type in their location and the bot provides local election results

ORF came prepared with the then growing list of what they dubbed ‘fall through scripts’ that triggered chatty responses to questions outside of the list of commands. Responses to spontaneous questions such as ‘Who will win the presidential elections?’ or ‘Do you think candidate A is sexy?’ sometimes included behind-the-scene video or even GIFs to reward the users who were trying to engage with the chatbot.

At Politibot, the team trained their NLP models using conversations from the Spanish spoken in Chile (on top of the Spanish language Politbot was already trained in) to adapt the bot to cover the Chilean elections. According to the team, this allowed the chatbot to recognise regional slang terms — including insults. For example, as soon as violent language was detected, the chatbot was able to engage with the user with expressions from the appropriate region.

Keefe told us the team at Quartz is still trying to come up with ways to optimise unexpected questions or responses from users. Quartzy — their cultural and lifestyle chatbot which is available on Facebook Messenger — no longer responds to smalltalk such as ‘How is your day?’ or ‘How are you?’ because it was making the user experience too confusing, according to Keefe. The team is taking a step back while they figure out how best to integrate chitchat back into the currently command-based chatbot. For the time being, Quartzy replies with the slightly irritating ‘Yeah, I’m not sure what you’re saying here’, which Keefe admits might make some users lose patience.

2. Focus on experiences rather than cramming content

ORF’s Naderer questions whether breaking news and daily stories are the way to go for chatbots.

‘Personally I don’t like that most chatbots are used as a boring newsletter-like tool. Newsletters don’t have a personality, but good chatbots do — even if they are command-oriented. Do you chat about breaking news with friends on WhatsApp? I never did, so maybe that use case doesn’t fit a chatbot. But we do share short and funny stories and the positive aspects of our daily lives, so maybe this is the way to build a good chatbot’.

Quartz are doing precisely that with Quartzy. While the Quartz news app already uses a conversational interface — Keefe tells us that the bot team at Quartz is now trying to go one step further by creating chatbot-exclusive experiences.

‘The experiment we’re doing with Quartzy was surprisingly effective. It’s true that if we just try to feed our content into Messenger, people might just say, ‘Well why do that? That’s not how I use Messenger.’ But if we use the interface the way it’s designed and the way people use it now [to chat with friends and family] then there is some opportunity there’, said Keefe. Currently, Quartzy can talk the user through how to make ‘no-knead bread’ in a fun and friendly way. The bread making experience lasts about two days and once you’ve completed the first step, Quartzy will ping you via Facebook Messenger around 12 hours later to remind you to complete the next step.

‘That’s not something you can do on a typical website or in any other formats’, said Keefe. ‘Some people actually made loaves of bread and sent us the pictures’.

At Quartz, the power of chatbot notifications was also explored throughout their ’12 Days of Sanity’ challenge where subscribers received a snippet of an inspiring story or a reminder on how to bring mindfulness to their lives. The chatbot exchanges were short and supposed to mirror the brief drop-in drop-out conversations a user would have with their friends and colleagues via messenger.

‘Our preliminary research through these experiments shows that those who opted in really opted in’, said Keefe.

But bot creators need to be very careful not to irritate their users.

‘One of the things we’re seeing in life is notification fatigue’, said Keefe. ‘As journalists and folks that are interested in the news, we have a lot of our notifications turned on all day and it can get a bit irritating.’ Rather than focusing on breaking news, which journalists might receive through push notifications or various other alerts, Keefe wonders if the bot is better off when used as a way to send friendly reminders.

‘I think that a lot of news organisations are probably moving away from chatbots is because they haven’t found them to be super successful. Maybe it’s just not the way people are going to get their news’, said Keefe.

But maybe it can be the way. Keefe told us that the CNN and ABC News chatbots seem successful. Politibot also sends out newsletter-style daily digests to its users through Telegram. And it’s not doing it in vain: Suárez and Biraud told us reactions were extremely positive.

‘Politibot saves you time: it keeps you informed about the really important stuff without having to deal with the unimportant stories that fill many news outlets nowadays. We send out a daily conversation where we explain a news topic with charts, links, pictures, and GIFs. Users always learn something they didn’t know and most of them get to the end of the text’, wrote the Politibot team.

3. Get personal — but not too personal

Unless you’re trying to create a companion or a digital version of yourself — such as the somewhat scary Replika — a chatbot doesn’t have to be all that human-like. There’s even evidence that people get ‘freaked out’ by all too human responses. According to Keefe, what chatbot users look for in a chatbot is a certain level of intimacy, which is not unlike what people want from radio or television.

‘I spent many years in public radio in the U.S. The talk show hosts and newscasters are people that you hear every day whether it’s in your co-ride home from work or in your kitchen while you’re making dinner’, said Keefe. ‘You almost develop this kind of one-on-one relationship with them. In the U.S., public radio really thrives from this level of intimacy, as the business model is people donating money to a service that they are getting for free.

‘At the same time, there is no illusion that the newscaster or talk show host is talking to just you. You know its a broadcast, you know that it’s news that’s going across the entire country. But you feel that it’s personal’.

It’s the same for chatbots. While the user knows that the CNN or Guardian chatbot weren’t exclusively made for them and anybody can chat to them on Facebook, it still takes place within the medium that might normally be reserved for family members and friends.

4. Create a personality

While newsletters might work for one chatbot but not so much for another, one thing that the team from Politibot, Naderer, and Keefe can all agree on is that tone of voice is key.

‘The success of our digests has to do with the intimacy of messaging apps like Telegram or Messenger and also with the conversational tone’, wrote the Politibot team. ‘We humans are wired for conversation. Chatting about the news with Politibot could be more rewarding and playful than reading a news article’.

The team worked a lot on the tone of voice of the bot before launching in June 2016.

The Politibot chatbot has personality

‘Politibot is intelligent and knowledgeable. He can be a little clumsy, but he is humble enough to admit a mistake. He is also a huge fan of Teddy Roosevelt’, says the team. ‘Our experience shows that building a consistent voice is important. You are not building a news article but a conversation and you have to build some kind of emotional connection with your audience. You can do it through humour or though cultural references. You can also do it by being useful or by explaining things they don’t understand’.

The Quartzy bot also has a strong editorial voice. Keefe told us that Emily Withrow, Quartz’s bot studio editor, wrote the entire script of the Quartzy chatbot. ‘She’s got a real sense of it — it’s super engaging and it’s full of imagery so that the people who came and participated went through the entire experience.

‘The key to a successful bot is actually really good humans, which is kind of amazing and beautiful’, says Keefe.

Screenshot from chat with Quartzy

4. Get the kids (under 35) on board

The Politibot user base is extremely young. More than 70 percent of users are under the age of 34 and there are more men than women.

‘All of our users are people who are interested in politics. They love reading books. They are clever people who are interested in foreign affairs’, the Politibot team told us.

Naderer also says that the Wahl-bot audience was younger than the average on their more traditional channels.

‘But this is normal for chatbots where the potential audience of your bot is defined by the user base of the Messenger platform you are operating on. The Wahl-bot was available only within Messenger, so its users reflected the service’s community’, said Naderer.

5. Make your chatbot profitable

Naderer is cautiously optimistic about the profitability of chatbots by reminding us that you can’t reach a huge audience with a chatbot, but rather a very special niche.

‘Maybe absolute numbers are too a dominant factor in our industry,’ Naderer said. ‘Looking at the raw numbers, bots are a small and profitless field, but their side effects are pretty impressive and will emerge in the long run’.

ORF built their Wahl-bot from the ground up without the help of a chatbot builder: They wrote the chatbot code, the library to communicate with Facebook’s API, and a lot of glue code to integrate existing internal content APIs. Not using a chatbot builder enabled them to get a deeper understanding of how the whole technology works and where its real limitations are.

‘We took the hard way and I’m positive this makes our learnings more valuable for future projects even if they have nothing to do with a chatbot. We improved how we generate our election website in the backend and we now have better validity checks for the incoming election data streams.

‘Most chatbots will never generate profits, but their side-effects can. […] So maybe you should think again to see if it really was a good idea to stop your chatbot project’ said Naderer.

Politibot are on their way to growing a business out of their chatbot and they are sustaining it by a global membership model. On top of chatting on telegram and Messenger, the team have also launched a podcast and signed a collaboration agreement with Random House Spain. Throughout 2018, Politibot will introduce sponsored content and talk about some of the publisher’s non-fiction books in its daily conversations with users.

‘We’re pretty critical people, were not building chatbots as it’s just some hobby. We actually saw some pretty exciting possibilities so we’re playing with that and see what else we can come up with’, concluded Keefe.

Some chatbots to chat with or keep an eye on:

Brazilian fact-checking organisation Aos Fatos are creating a Messenger chatbot called Fátima aiming to provide users tips and resources for debunking misinformation. Read more here.

Quackbot: slack bot for journalists; grabs screenshots and points out cliches.

Hipster Catbot: type in where you live and the catbot gives you suggestions about where to grab coffee. Useful, except for the moment it told me that I was looking fabulous today.


John Keefe is Bot Developer and App Product Manager at Quartz. He is also adjunct instructor at CUNY graduate school of journalism. He previously worked as senior editor, data news at WNYC Radio.

Philipp Naderer is web developer at ORF. He has also built a chatbot called Seestadt Bot, which is one of Austria’s first chatbots.

Eduardo Suárez is the co-founder of Politibot. He is also reporter at Univision Communications Inc.

Miguel Eduardo Gil Biraud a freelance software developer. He is chief of technology at Politibot.