What we learned from a WhatsApp chat during a GOP presidential debate

Guardian readers loved the format, but the platform seriously lagged.

We’re excited to tell you about our first experiment: a live WhatsApp chat during the Republican presidential debate on December 15, hosted by Adam Gabbatt, a politics and culture reporter for the Guardian US. Below we outline how we decided on the project and set it up, some basic metrics, what worked and what didn’t, and our recommendations both for newsrooms considering using WhatsApp in this way and for WhatsApp itself. Read on, and let us know what you think.

Why WhatsApp?

In starting up the lab, we wanted to get going quickly with simple experimentation, to learn by doing and to gain familiarity with working within the Guardian US newsroom. We met with Lee Glendinning and Matt Sullivan, editor and deputy editor respectively, and brainstormed experiments that could be done quickly and easily around the next big news event, the GOP debate.

Soon we landed on the idea of a live chat in WhatsApp, for a few important reasons:

The user base is massive. Chat apps, with their huge numbers of users (WhatsApp announced in September it had reached 900 million monthly active users), offer news organizations a big opportunity for reaching new audiences on mobile and for communicating with them in new ways.

No major newsrooms have tried a two-way chat. Or none that we know of, anyway (but please let us know if we’ve missed one). Those that have run projects on WhatsApp have mostly used the platform to send updates about a news event, or to gather user-generated content. For more on some of those cases, see this recent report from the Tow Center for Digital Journalism.

Chat apps facilitate direct and informal connection between readers and journalists. One of the biggest strengths of chat apps — the capacity for one-to-one connection — had yet to be explored in the context of a news report. So we wanted to see how an audience would respond to a journalist reporting and responding within a small-scale chat.

The debate had built-in parameters. The topic promised an interested audience (those following politics), and the event had a clear beginning and end.

The setup

Setup was easy, and involved five simple steps.

1. We designated a phone in the office as the WhatsApp phone.
2. We created a new broadcast list in WhatsApp.
3. We had a handful of meetings with editors, Adam and the audience development team to discuss our approach to the chat content and a promotional strategy.
4. We drafted an article inviting people to join the chat.
5. We created a Google form survey to send out at the end of the chat.

The execution

Preparing: We posted the chat invitation as an article on our website describing what we planned to do, with instructions on how to sign up. In writing up the invitation, we drew from examples of previous experiments done by the Globe and Mail in Toronto and by students at Birmingham City University in the UK, as well as by our colleagues at Guardian Witness.

The audience team posted the invitation to Twitter and Facebook. Once the liveblog for the debate was active, its editor added links there. The bulk of sign-ups came in just before the debate itself, and mostly from the liveblog, not the social posts.

We decided to limit the chat to one broadcast list — WhatsApp’s term for a list of subscribers, to which one person (the host) can send out updates to everyone subscribed. Subscribers can chat back individually, but not to each other — a one-to-many-to-one model. WhatsApp limits each broadcast list to 256 members, and we reached that number around the start of the debate. More than 100 people asked to join after we had reached the limit.

(The alternative to the broadcast list would have been a group chat, which would have had the advantage of allowing all participants to chat together, but also made their phone numbers public, a clear privacy minefield. It also limits the number of participants to 100.)

We created only one broadcast list to keep the chat as manageable as possible for Adam, to keep the administrative aspects feasible, and so we could better follow the posts and the audience’s responses to them.

Chatting: Adam posted a mix of observations, a humorous short video and a few photos to the list for several hours, as he followed the debate from the screen and on Twitter. He was able to write back to some of the chatters, though less so as the debate progressed and his attention was on keeping up with the debate and relaying his observations. Still, he did have time to acknowledge some reactions and also to post them to the group, along with videos and photos. (For more about the quality, see “What worked” below.)

Meanwhile, we supported him by maintaining the list — removing the few who asked to get off, adding others — and also suggesting formats, taking video and creating a gif, so that he could focus on posting updates.

Once the debate was over, we posted a link to a Google form in the chat list asking for demographic information and, more importantly, qualitative feedback on the live chat experience. Some of the results are below.

Basic metrics

Chat stats:

• 62 total chat messages (57 texts, 3 pictures, 1 video, 1 gif, 2 links)
• 4:40: length of the chat (7:01pm to 11:41pm ET)
• 394 total sign-up requests
• 275 participants (original 256 + extras, after drop-outs)
• 119 people who couldn’t join since we were at capacity
• 98 participants chatted to Adam directly
• 22 participants received a direct chat back from Adam
• 20 participants asked to be removed from the list

Survey results:

• 52 total responses
• 70% of respondents use WhatsApp every day or a few times a week
• 20% of respondents reported using it rarely or while traveling
• About 70% of participants used the chat as a second screen to TV or online streaming
• About 70% of people were watching from home; another 15% were at a bar, restaurant, friend’s house or in transit

What worked

The intimacy and informality. The small, closed nature of the chat helped bridge some of the usual distance between reporter and audience, and worked well with an informal tone. In addition, Adam said he felt a similar ease of chatting as he has when talking to friends; it helped that he is a heavy WhatsApp user and was already comfortable using the platform. Judging from the positive feedback in the chat, chatters seemed to enjoy the sense of closeness to a reporter and the informal tone. Among the responses we received:

“Felt more personal than Twitter or the comments section.”

“Great job of reaching the audience. Tone/syntax was perfect, made reading through it easy as if reading texts from a friend reporting on a big event.”

“The humour was great. The feeling of being involved in a developing story was quite exciting.”

The tone. Adam is a heavy social media user and he recognized the opportunity to make the chat a group experience. He was able to quickly establish a welcoming tone and communal feel, despite the one-to-many-to-one structure of the broadcast list. He asked chatters to help him name the group, giving it a special status (he settled on “WhatsApp squad”) and referred to what participants were saying throughout.

The quality of user responses. He also found participants’ responses to his posts tended to be substantive and also useful in his reporting. Some sent in their own reactions to the debate as well as links to related stories, fact-checks and photos, a few of which Adam used in posts.

What could have worked better

Many of the challenges arose from the mechanics of WhatsApp. The platform was built for individuals, not publishers, so it doesn’t include a full set of tools geared to a newsroom’s needs. Use of the broadcast-list format is a workaround on a platform with limited functionality, and proves to be time consuming and somewhat messy.

For example:

Adding and removing subscribers. Adding new list subscribers was a challenging, multi-step process. Removing them was even more labor-intensive. It was sometimes hard to see who had asked to be removed from the list since the individual chat screen would be updated each time Adam posted something new, making it necessary to go into each person’s chat and try to find the word “stop”. We actually missed a request until we reviewed all the messages the next morning, leaving one subscriber with a very poor experience. (The New York Times found this out the hard way when a recipient of its posts about Pope Francis’s trip to Latin America turned out to be a fellow journalist who wrote about it.) WhatsApp simply doesn’t include adequate functionality to make those requests readily visible, not to mention automatically fulfilled.

Web-based limitations. In addition to a phone interface, WhatsApp offers a web-based interface, WhatsApp Web, that can be open at the same time as the phone chat. The functionality of the web version is limited to chatting, whereas account and list maintenance must be done by phone. Adam found it easier to post text via the web, but to post video or photos from the phone. When used as a publishing tool, the lack of access to multiple web windows and the inability to manage accounts from them is bit of a limitation, as it prevents, for example, two or more staff from working simultaneously.

Lack of subscription confirmation. Although many participants had used WhatsApp already and had no problems with sign-up, others were uncertain when they didn’t immediately get a confirmation. One user gave this feedback: “I thought I wasn’t ‘in’ although it wasn’t clear to me.” We mentioned in the sign-up instructions that we wouldn’t be sending any updates until shortly before the chat. Still, some weren’t clear about what was happening, and texted “JOIN” repeatedly. In retrospect it would have been a worthwhile courtesy to confirm, although we would have had to repeat the additional step 256 times.

Unwieldy individual chat windows. Each user’s response to the host triggers a new chat window. When you’re chatting with 256 people simultaneously in a broadcast list, it would be possible (though unlikely) to have that many chats going at the same time. Even when there are only a fraction of that number, the experience for the journalist is unwieldy. Adam did read many users’ reactions and posted a few within the chat, but didn’t have time to post and respond to everyone, which may have frustrated some users.

In addition to mechanical or platform-related challenges, it was clear that participants wanted to experience a group chat. The overwhelming majority of responses to our question about what we could have done better reflected the wish for it to be a group chat. Participants wanted to hear not just from a journalist but also from others around the topic of the debate. They responded with “see people’s comments” and have “better interaction with other people”. One noted: “It was a great feeling to know a Guardian reporter was on the other side, but felt like I couldn’t take full advantage of it. Couldn’t exchange views, have real small chats about the candidates or topics being discussed.”

A few conclusions

Scaling informal, intimate interaction is difficult. While the intimate format allowed for meaningful interaction between audience and journalist, the limitations of the platform made it hard to manage. The possibilities for building community are promising, though, and we’d love to see other kinds of newsrooms experiment with it, particularly at the local level or within a niche.

Publisher-friendly tools seem to be required for replication. If chat apps were to lower the barriers to entry by launching a set of publisher-friendly tools, it might become a viable and powerful way to communicate with audiences and build community. Without those tools, it’s difficult to imagine a scenario in which this kind of experiment could be routinely replicated and done at scale.

Risk of alienation, but may open door for membership. Another possible drawback is that broadcasts and chats are essentially private clubs, and their use risks alienating some who would like to participate. This exclusivity may not be right for a lot of newsrooms, but may be an opportunity for organizations requiring paid subscriptions, with participation becoming a benefit for members.

Recommendations to newsrooms

Our best advice for covering a live event on WhatsApp:

Think of a WhatsApp project as experimental. The platform is not built for publishers’ needs and doing anything at scale would be difficult.

See WhatsApp chats as a chance to communicate with readers and offer them information on a platform some are likely to be using already. Don’t expect traffic.

Plan well beforehand. Discuss timing, invitations and who will handle what. Test out WhatsApp functionality if you don’t use it regularly.

Give consideration to formats. Will what you’ll post overlap with coverage elsewhere — in a liveblog, for example? Will efforts be duplicated? Would it make sense to share content on one with the other?

Post clear sign-up instructions. Be as transparent as possible about what potential participants can expect, how they can unsubscribe and whether or not you plan on messaging them after the project is over.

Be ready for a bit of craziness when managing sign-ups and drop-outs.

Have at least two people working on the project. For the debate, ideal staffing might have been: one chat host, one person to manage the list and to pay attention to responses, one to make gifs and videos, and one to support the host. Be ready to share responsibilities. Above all, communicate a lot during the project.

Create a few images and videos beforehand if possible so you have a few ready to go. The pace can be rapid during a live event, making it hard to pause even long enough to take a quick photo or video.

Involve the audience. Ask questions and let them know you’re hearing them, either with direct responses or by referencing them in posts. Chances are you won’t be able to acknowledge everyone. Do what you can.

Think through what you’ll do at the end. Send a clear message when you finish; include information about any follow-up you may do in the future.

Consider what you might do with the list afterward. You now have direct access to a tiny yet actively engaged segment of your audience, and the opportunity to create a micro-community. Are there other ways they can participate? Are there related kinds of content to which you can point them?

Recommendations for chat platforms

Create a web-based console for easy administration of sign-ups and stops, so more time can be spent moderating and participating in the chat, rather than on list management.

Create a professional tier account option that offers more tools and functionality for the chat host, including a searchable database of past chats and participation metrics.

Offer personal privacy controls for chat group members, allowing participants to create semi-anonymous avatars that give them an identity in the chat but don’t reveal their phone numbers.

Provide support for line breaks in messages, eliminating the need to type text in a separate program and then copy into the web interface.

Make it easier to batch export messages instead of sending one chat at a time, for easier analysis of quantity or sentiment of responses.

Allow more flexibility with the number of participants in each chat, so groups can be the appropriate size for the event.

We’ve posted some of our project documents publicly, for reference. Included is all the media uploaded into the chat, a summary of the Google form quantitative feedback and screenshots of the most interesting chat responses. Nieman Lab, our partner in distributing information about the mobile innovation lab and its findings, also wrote about the chat in this recap.

We’d love your feedback on anything related to this post, particularly if there’s other info that might be helpful to you. Leave a comment or email us innovationlab@theguardian.com.

And if you’re in a newsroom and have tried out a chat app, we’d love to hear about it. What worked? What didn’t? Would you do it again?

— Sasha and Sarah