Web notifications for news: Lessons from the jobs report, round 3
We’re continuing to test features of interactive web notifications by using them to cover the US jobs report.
On July 8, we ran a third test of interactive web notifications, which give users the opportunity to move through a set of information using action buttons, for the monthly US jobs report. We’ve been studying notifications as one of our five areas of focus, and ran our first experiment, about the April US jobs report, in May, with an audience of only our colleagues in the Guardian US newsroom. In June, we opened the experiment to the public, inviting readers of the Guardian’s business blog to receive notifications about the May jobs report.
This most recent month’s experiment is a continuation, in which we looked to further test hypotheses about how users interact with notifications, and challenge some of the early data we collected.
What stayed the same
The same team of Guardian Mobile Lab and Business desk reporters and editors worked on the project this month. We decided to stick to the Good News/Bad News paradigm we’ve described earlier, including the new 😀 and 😟 emoji. We held one meeting in advance to plan and then wrote the script for the notifications together, using commenting in Google Docs to answer questions about content as they arose.
What changed
Each month we try to test and learn at least one new thing about our formats. This month, we randomized the the action buttons, to avoid always having Good News showing on the left and Bad News on the right, which we suspected would privilege one over the other and give us slanted results. Also, after issues we had last month with notification delivery issues, we were looking for more information on how many times our notifications were shown. Lastly, we sent a notification containing a survey following the experiment, something which has become standard practice for us in other notifications experiments but that we hadn’t done yet with the jobs report audience.
Audience
This month, following the success we had in recruiting users to our Brexit experiment via the sub-headlines of the live blog (known at the Guardian as standfirsts), we recruited subscribers through links in the Guardian’s Business live blog during the two days leading up to the jobs report release.
We eventually recruited 340 subscribers, up from 150 subscribers in the month previous. In fact that number included 50 people who had received jobs report notifications the month before, and had opted in through a preemptive notification we sent them asking if they would like to be a part of this month’s experiment as well.
Results
We still have issues tracking the delivery. While we had 340 subscribers, analytics reported that the initial notification was only shown 202 times. This is far better than the previous month, when only a third of the notifications sent were shown.
Of the 202 notifications that were shown:
- 60 users (30%) tapped directly on the initial notification and went to the live blog
- 46 users (23%) immediately dismissed the initial notification and did not interact further
- At least 38% of users who started the series made it all the way through the series of notifications
- 23% of users who started the series opened the live blog or the article after seeing all the notifications
- 44 users (22%) started the series by tapping on either a Good News or Bad News action button
Survey data told us…
Since the last jobs report experiment in June, we have sent links to surveys out through a last notification after each experiment, seeking qualitative feedback to supplement quantitative data from Google Analytics. For this experiment we sent the survey around 1 pm the same day, and received 58 responses. Some findings:
- Users are by-and-large interacting with the notifications in the intended environment. 87% of survey respondents viewed the notifications on mobile.
- Users understand the Good News/Bad News paradigm. 62% of respondents said that splitting the news into good and bad news made sense. An additional 29% said it “kind of” made sense.
- Users found the notifications informative and interesting. 86% of survey respondents rated the notifications in the two highest categories when asked if they were interesting. 77.6% of survey respondents rated the notifications in the two highest categories when asked if they were useful.
- Some users think the Good News/Bad News buttons are hard to find. 27% of respondents said they did not think the action buttons were easy to find. An additional 15% percent said they were “kind of” easy to find.
- Some users never got the actual notifications, but did get the survey. A few people who responded to the survey told us they never received the initial notification. We know there is a gap between the number of notifications we attempt to deliver and those that are seen by users. Our hypothesis for this particular incident is that the randomization of the action buttons between good news and bad news may have caused additional delivery failures. For those looking to experiment by adding small iterative new features each time, this can be a lesson to spend extra time testing new complex actions within a notification and try to anticipate issues that may arise when you scale from internal testing to external delivery.
- Users who do follow the jobs report said this experience was worthwhile, and those that do not follow the jobs report said it made them more interested in the topic.
- Users said that they would expect to follow this as a series. As a means of testing assumptions of when we need to send additional opt-in alerts to users, we asked if they would expect to automatically receive notifications on next month’s job report. 69% said yes, that would makes sense to them.
What we learned:
With randomization in place, users had no preference towards good news or bad news. One of the most interesting things we learned this month is that when the placement of the good news and bad news button is randomized, users do not show a preference of which they choose to see first: 23 of our users tapped the button for good news first, and 21 tapped the button for bad news first — a near even split.
Adding explanatory text to the numbers requires more editing. Each month we have added more text to each notification series in an attempt to make the content more compelling, clear and useful.
We start by writing a document with copy for each alert, divided into the Good News and the Bad News paths, and with “TK” to stand in for numbers that will be contained in the report. We also add a few extra backup alerts at the bottom of the document that could be categorized as either good or bad depending on the jobs report numbers.
This is a good start, but as we have increasingly tried to not just state the facts, but also add context to the news with a sentence or two of explanation (what a change in the number of part time workers implies about the health of the economy, for example), it has taken longer to compose alerts with new report-based information in a way that fits into our character limit. This month, it took nearly an hour to write and edit alerts before we could send the series even with the document preparation. This delay was called out by our survey respondents, some of whom clearly expect quicker updates.
It’s uniquely difficult to collect reliable analytics for interactive notification series on mobile web. When we send the user a chain or series of notifications, the entire chain immediately gets downloaded and the user is able to access the entire experience. However, getting analytics back on the way they interact has proved more difficult. When they tap, a network request is made to send back analytics data. If a user is in a connectivity deadzone, that data won’t get sent. So to accurately assess whether or not a user saw or interacted with all seven notifications in the series would require that they had near-perfect connectivity each time they interacted with a notification or an action button.
For this reason, we can say with certainty that 17 of our users saw the full notifications series, but also that the number could have been higher. As we continue to experiment with interactive notifications, developing a tagging system that will give us fuller insights will be important.
If you tap on the first notification and go to the live blog, there is no way to re-start the series. At least 60 of our original users tapped on the notification immediately and went to the live blog. This may have been intentional on the part of the user, or it may also be the case that they had wanted to go through our notification series but didn’t realize how. For someone who either has gone through the series and wants to replay it, or missed the functionality entirely we currently do not have a mechanism for users who want to replay the notifications. We’d like to find a way to allow users to restart a sequence if they have missed something, or simply want to replay it. We’ll be exploring ways to do this in the future.
Looking ahead
This coming month will probably be one of the last times we will run the jobs experiment in the Good News/Bad News format. Next month, we plan to test implementing a safeguard against the randomization of buttons causing delivery issues
Looking farther ahead, one of the new types of notifications we have been developing (particularly in advance of iOS10) is a quiz format …. stay tuned.
As part of our study of notifications we will be writing up our ongoing jobs report experiment as a monthly series to explain how we are iterating on the format. Our next experiment will be on August 5, the date of the release of the July jobs report. The intended experience is for Chrome on Android devices. Simple sign-up is available on our website.
Let us know if you have additional ideas and observations in the comments or email us: innovationlab@theguardian.com. We look forward to hearing your thoughts.
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
