We made a Facebook Messenger bot to track people’s feelings about the election. Here’s what we learned

TL;DR: There is value to treating messaging bots as a primarily in-app experience

By Teddy Amenabar, Chris Ballard, Kayla Epstein, Sophie Ho, Joseph Price, and Shalini Pusapati

During the final weeks of a volatile election season, The Washington Post launched a bot designed to track the emotional ups and downs of each day. Feels bot was intended to monitor and compile data on how various shifts in the news cycle were impacting people.

We also wanted it to be a place where users could safely express their opinion and hear from others — including people they may not agree with.

Every night, we asked readers to tell us how the election was making them feel that day. Readers were allowed to respond using one of five basic emoji — 😄 🙂 😐 😢 😠 — (an emoji set that we could render consistently across all platforms) and given the option to elaborate with a sentence or two.

The next morning, the bot would deliver a report showing a graph of reactions and highlighting several quotes from other users, carefully selected to showcase the most interesting and most enlightening responses. Here’s what it looked like in action:

Mockups show the bot in action. Left: the Feels morning report. Right: Feels user responses, and the evening survey. (Chris Ballard/The Washington Post)

After the election, each user received a personalized report of all their reactions throughout the duration of the experiment.

This all seems straightforward — and we wanted the functionality to feel simple for users — but the bot was the product of months of brainstorming, building and testing.

WHY THIS APPROACH?

We were particularly keen to use Facebook Messenger, as we’d launched several bots this year and wanted to apply what we’d learned. In particular, we were eager to try an approach that engaged with users rather than simply delivering news to them. After all, there was no shortage of other places for users to read up on the election.

We also wanted to apply lessons we’d learned from other community- and engagement-based projects around the election. We knew that creating positive spaces for people to talk about politics is a difficult task and were looking for an approach that would create that interaction and exposure to other people’s viewpoints, but in a highly directed way. Messaging bots, in general, provide a systematic way to talk one-on-one with people. And by disseminating some of those one-on-one interactions back to the group, we hoped to create a light sense of community among users to encourage ongoing participation.

HOW WE DID IT

Conceiving and building Feels was a true collaboration between teams in the newsroom and in engineering — product, design, audience, and community all played a role.

Refining the concept: To show users their emotional journey over time, we needed to be consistent in what we were asking them and how we presented information back to them. After playing with and soliciting feedback from our pollster on several variants of a daily question, we decided to ask readers, “How is the election making you feel today?” This phrasing allowed the question to be consistent, but for the readers’ answer to be variable, and dependent on the events of the day.

Integrating design and text to encourage participation: We were keenly aware of the challenge in getting users to actually engage with the bot every day until the end of the election. We also needed to onboard people who were not familiar with using “bots” or Facebook Messenger.

We looked for ways the delivery and tone could be improved with images, emotion and personality. We gave the bot a personality by naming it “Feels” and creating a robot caricature to act as the bot’s “face.” All dialogue and potential conversation threads were mapped out to guide users through the experience, anticipate questions users might have, and counter attempts some might make to “break” or troll the bot.

A section of the script we wrote for Feels. This portion shows the flow the bot would go through if the user wanted to share more information about their emoji or not.

For example, when someone tapped Get Started, Feels was there to greet them with a simple onboarding process. When someone asked for help, an image of Feels’ face popped up with the menu of options. If someone sent it profanity, Feels would respond with disapproval.

Readers quickly learned what Feels could and couldn’t do. If you tried to “break” the bot with a question or comments, Feels would just say it doesn’t understand and, eventually, the bot would stop responding.

Building a backend as well as a frontend: We needed an easy way for us to receive, edit, and send out user responses once we got them using the bot.

Thanks to our previous bot infrastructure, we were able to get up and running fast. But, we quickly learned that with our focus on push messages and user prompts — messaging lots of users all at once — we had to redesign our system for performance and scale. We also made sure that the architecture was flexible so we could add new paths and features based on how users were actually engaging with the bot during our testing phase and after launch. We smoothed out some sign-up and unsubscribe functionalities post-launch. We also tweaked how we messaged users who signed up after 7pm so they still got evening question.

We also developed a simple content management tool that allowed editors to manually compose and send those morning and evening messages to all users. To make it easier to see all the users’ feedback, we channeled responses into a Slack channel, where we could easily scan feedback and select the items to share in the next morning’s report.

SO…WHAT HAPPENED?

Engagement among our users was gratifyingly high throughout the bot’s run. A third of users participated every day, and the user base grew by 11% daily. A full 85% of users stayed subscribed throughout the entire experiment.

In the early days of Feels, users reported feeling mostly 😐, for a wide variety of different reasons. But we did see our readers’ emotions change around some significant points in the race.

This gif shows how reactions to the election changed on key dates (Chris Ballard/The Washington Post)

The clearest example came when Hillary Clinton’s emails were surprisingly thrust back into the spotlight in late October. The reaction to the news was immediate. Our Feels users went from being mostly 😐 to predominantly 😠.

Here’s a chart that showed how their reaction changed after the email news broke:

You can see the 😠 emoji peak on October 28. (Joseph Price/The Washington Post)

We also shared various reactions to the email story among our users, and the nature of the bot allowed their opinions on a controversial subject to be heard without instigating an unhealthy conversation among the group.

This was exactly what we’d hope the bot would do: reflect how people felt about the election in real time and expose people to differing opinions in a constructive way.

WHAT WE LEARNED

We had a few takeaways from this bot experiment that we’re eager to apply moving forward.

Our main realization was that there is value to treating messaging bots as a primarily in-app experience. We did share some links to relevant news stories with our users, but we never expected Feels to be a great tool for generating traffic. And it wasn’t. The goal for something like this has to be providing readers with a way to participate where they are, and we saw that the users who participated felt like they were getting a satisfying and complete experience within the app.

One challenge that became pretty apparent over the course of the experiment was that the average user doesn’t know what a “bot” is. It’s an industry term that’s not commonly understood. In addition, although we called Feels a “bot,” in terms of utility it was closer to an app that happened to be hosted on Facebook’s Messenger platform. Some users expected Feels to carry a conversation similar to an AIM bot. The more specific and straightforward language we can use to set the correct user expectations about an experience, the better.

“Bot” also implies a great deal of automation, but in reality our team did much of the work manually — and it took a great deal of time. Curating the responses we received and crafting the messages we sent out was labor-intensive. Ideally, it would be nice to automate more of that work, but it is more important to be careful about the quality and variety of the responses we shared back with the community and automation couldn’t have provided that.

Overall, we see a lot of potential for Facebook Messenger projects that focus on engagement. Possible applications could include community-building around other news events with emotional impact, letting users discuss controversial issues safely, and collecting quality user-generated content.


Got more questions about Feels or Facebook Messenger? Leave them in the comments below and we’ll try to answer them for you!

Did you like this article? Hit the 💚 to recommend it to friends.