Web Notifications for News: Lessons From the Jobs Report, Round 4
In which we sunset the Good News/Bad News paradigm.
On August 5, we ran a fourth test of interactive web notifications, which gave users the ability to move through a set of information using action buttons, for the monthly US jobs report. We’ve used the same format each time we have covered the jobs report: breaking the jobs report down into along the lines of “good news” for the US economy, and “bad news.”
Having solved some enduring technical issues and gotten the editorial workflow down to a routine, we’ve decided that this month was the final time we will use the Good News/Bad News paradigm as a way to cover the jobs report.
One of the reasons we liked the idea of using the US jobs report as an experimental testing ground is because it is replicable on a monthly basis, allowing us to test hypotheses against slightly changing variables each month. Our closing down this experimental format means we are opening up space for a new type. And so starting next month, we’re going to switch it up (stay tuned for more details).
Over the last four months, we’ve recruited a group of about 350 users. Our first public experiment using the Good News/Bad News format was in June, covering the May jobs report. We recruited users through Twitter, and through the business live blog the morning of the experiment, eventually reaching 150 subscribers. The next two months, we repeated the experiment and also recruited similarly, through the live blog starting the day before the jobs report was released.
We’ve learned a lot from this sort of replicable experiment. Some of what we’ve learned is about our own procedure and primarily of use to our team. Other things, like those below, we think offer important takeaways for a newsroom who wanted to try their own version of an interactive notification.
Breaking news using an interactive notification requires a subject expert. Editorially, our first takeaway is to say that if you are planning to run a complex experiment on a dense topic in real-time, you need the help of a reporter or editor with in-depth knowledge of the subject. We have been so happy and fortunate to work with Jana Kasperkevic, business reporter at the Guardian. Jana has been there for every jobs report experiment and has been our primary source of information as we write the copy for the notifications. She’s been there to explain what the fact that the interest rate hasn’t risen this month means about consumer’s and the Fed’s confidence in the economy, and there every time I have started a comment in Google docs that starts, “Is there a simple way to say how….” (the answer is usually no).
Yet, as we have made the notifications progressively more editorially complex, our time from getting the information in the jobs report to sending the alerts distilling that information has increased, despite the fact that we have developed a routine. The reason behind this is because we don’t have a subject expert on our team, and we are reliant on Jana to help us interpret the news as it emerges. For newsrooms looking to experiment with real-time interactive notifications, these also need to be staffed with a subject expert much as a regular shift would.
The default behavior is definitely still for users to tap on or dismiss a notification. And that’s ok! Our experiment has been in part to test the format and see if we can get users to anticipate that there is a different experience in the notifications. To that end, we tried two different types of written directions in the notification, first telling users to “tap below,” on the action buttons, and then telling them to “expand” the notification to show the action buttons. Around the May jobs report, we found that 32% of users did not start the series, for the June jobs report that increased to 53%, and for the July jobs report, the last round of experimentation, 58% of users did not begin the series.
The numbers won’t tell us if one language choice was more effective (or if people are even reading the directions). but we did have some users find the action buttons, as shown by the 22% of users who completed the sequence in May, and 11% of users who did the same in June. This suggests that yes, this new behavior can be learned.
Your notification format needs to meet user expectations… We originally designed the Good News/Bad News paradigm as a way to report some of the numbers below the top line “number of jobs created” figure. We decided on the good news and bad news as a way to share some of those numbers in context. For example, we highlighted what happened in particular sectors — like energy, month over month — and also looked at unemployment amongst different age sectors.
Over time, these became more editorially complex and more explainer-like. This also meant it took us longer to complete, test, and send, with the time-sensitive information. While we had alerted users that these notifications would be experimental, what we offered — an editorially complex and interactive package — was nonetheless not timely, something which our users told us in our follow-up surveys was important to them.
The lesson here: Newsrooms looking to experiment with new types of notifications should make sure that their goals for experimentation still align with the expectations of the users. In our case, users were expecting we would send our alerts at the same time or near the time the jobs report news broke, but our format didn’t allow for that.
..and how you recruit plays a role in what those user needs are. At the same time, our audience for these notifications was recruited in the Guardian business live blog, meaning that these readers were already at the least interested in business news, and extrapolating further, were likely fairly business savvy. Results from the surveys we sent out after the notifications experiments support this: Only 8.9% percent of our survey respondents said that they did not follow economic news.
However, our format, giving the good news and bad news, became more explainer-like over time, trying to tell the user what these particular numbers meant, in context. Frankly, our audience probably didn’t need an explainer, and the increased time it took us to put these more editorially complex notifications together meant that we sacrificed what for them was their key need: speed. If we were to have recruited a less business news-savvy audience, our users might have appreciated the news breakdown, but as it stands, we were not serving the primary need of our users — and the responses they gave told us that.
As part of our study of notifications we will be writing up our ongoing jobs report experiment as a monthly series to explain how we are iterating on the format. Our next experiment will be on September 2, the date of the release of the July jobs report. The intended experience is for Chrome on Android devices. Simple sign-up is available on our website.
Let us know if you have additional ideas and observations in the comments or email us: innovationlab@theguardian.com. We look forward to hearing your thoughts.
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
