Web notifications for news: Lessons from the jobs report, round 2
We’re iterating on web notifications formats by using them to cover the US jobs report. Here are this month’s findings.
On June 3, we ran a second test of interactive web notifications for the monthly US jobs report. We’ve been studying notifications as one of our five areas of focus, and ran our first experiment, about the April US jobs report, in May, with an audience of only our colleagues in the Guardian US newsroom. This time, we opened the experiment to the public, inviting Guardian readers to receive notifications about the May jobs report.
Working again with Guardian business reporter Jana Kasperkevic and editor Dominic Rushe, we wrote a series of notifications to shed light on the underlying numbers released in the jobs report and the implications of what ended up being a dismal month for the economy. We carried out this test using much of the same process we had tested a month earlier, and learned a few new things.
An iterative approach
As we continue to test the web notifications format, we will continue to use an iterative approach, making changes to layout and tweaks in text to help improve overall effectiveness. For this second test, we re-used the format from our first notifications experiments. We maintained the good news/bad news flow, which gave users the option to follow one or the other in a sequence of alerts, and slightly changed the emoji, the initial notification text, and text we used in the action buttons to signal the choice between good news and bad. We had heard from some April jobs report subscribers that they thought the ‘thumbs up’ and ‘thumbs down’ emoji in the action buttons were asking them to rate the jobs report as good or bad, rather than recognizing it as a navigational element to move through the notification series.
This time, we changed the emoji from thumbs up and down to smiling face and frowning face and made the text in the action buttons more active: Give me [smile emoji] news; Give me [frown emoji] news. We also sought to add more context to the notifications text itself. Later notifications also included the word “more” to signal what the next notification classification would be.
Audience
We recruited users for this first public test through the Guardian Business Twitter account. Guardian’s business desk tweeted the invitation several times the day before the jobs report came out announcing our test and sending interested readers to a signup page. We eventually recruited 150 subscribers. However, due to technical problems with message timeouts which then caused delivery issues, our notifications were only definitively delivered to 50 of our 150 subscribers. It was a small audience, but larger than our previous internal test and, more important, external to the Guardian.
Results
Of the 50 who received the notifications:
- 35% of users started interacting with the alert sequence
- 28% completed the entire interactive sequence and saw all of the alerts
- 20% clicked-through to the jobs report article after completing the entire sequence
- Twice as many preferred to ask for good news first
First, 50 is not a lot of people, and we will have data we can draw deeper conclusions from in later experiments. Still, the fact that 28% of subscribers completed the entire interactive sequence tells us that people are figuring out how to surface and interact with the action buttons, and 20% were further incentivized to read the full article for further context and color after learning the top line news from our notifications. As we recruit larger audiences for experiments, we’ll monitor these engagement trends closely.
There were three big new takeaways:
There is a cut-off limit on text in notifications. As seen in the screenshot below, in which a notification accidentally got cut off, trying to add additional context in notification can result in too much copy. We’ve gleaned that the maximum number of characters in a title can be about 30, and that the body of the notification should have no more than 270. Because of size of some letters over others and the size differences with capitalization, these character limits vary slightly, but those numbers have served as our general guidelines.
There is a lot to learn about how people interpret emoji in text. Truthfully, it was a complete surprise to us that users would interpret the thumbs up/thumbs down emoji as a vote. But once we heard that feedback, and thought about it, yes, it definitely made sense. We’re conditioned by upvoting, or liking on Facebook, to think of a thumbs up as a way to say “I approve” through emoji.
While in our first experiment we asked “Do you want the good news or the bad news first?” in the notification text, we realized we needed to make the paradigm more clear, and also stop using those particular emoji. As we continue to experiment, we’ll be conscious of how these visual cues play: not just person-to-person, but also across platforms.
Let what you don’t get the first time guide you to the next iteration. For this experiment, working with only one developer (we now have two!), we prioritized setup and testing of our overall notification infrastructure over the implementation of more sophisticated analytics tracking.
We knew we would have an analytics gap but it became apparent after the experiment that, in order to truly assess a feature’s success, tracking each user’s interaction path from the beginning to end would be essential. So while our current data is accurate in the cumulative sense, having insight into each user’s path through the content would provide even more insights.
As we move forward with job reports experiments (we plan to keep running them for at least a few more months) we’ll have a set of questions in mind based on what we’ve done so far:
- Do users really want to choose the good news over the bad news, or will randomizing the placement of action buttons show us something different?
- What are other paradigms we could use to deliver interesting facts about the jobs report through notifications?
- Do people know how to reveal the action buttons below a notification? How can we increase familiarity with this behavior?
- What is the best way of showing the notifications to those who don’t see them in real time?
- The character limits we’ve found mean that titles need to be short but still descriptive. Is there a better way to do this?
As part of our study of notifications we will be writing up our ongoing jobs report experiment as a monthly series to explain how we are iterating on the format. Our next experiment will be on July 8, the date of the release of the June jobs report. The intended experience is for Chrome on Android devices. Simple sign-up is available on our website.
Let us know if you have additional ideas and observations in the comments or email us: innovationlab@theguardian.com. We look forward to hearing your thoughts.
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
