Profiles in Notifications: Lessons Learned From the Jobs Report, Round 6
Sending snapshots and receiving responses
October marked the sixth time we’ve covered the jobs report through notifications. To recap, up to that point we’d done four rounds of experimentation with interactive notifications and one month of a reader call-out through notification. For October’s jobs report we decided to change the format again.
We again worked with Jana Kasperkevic, a business reporter, and her editor, Dominic Rushe, to come up with a plan. Several months earlier Jana had mentioned that September 2016 would be the eight-year anniversary of the US economic crash — an entire presidency ago. Jana was working on a set of profiles of Americans, asking them how they felt they had fared over the last eight years. At her suggestion, we decided to adapt that piece into the October experiment.
Jana’s story, “The Great Recession, the hollow recovery” was published the day after the jobs report went out and included interactive elements and a custom format. The piece included excerpts of text and audio from interviews Jana conducted with five Americans whose lives had changed since the beginning of the recession. When combined, these excerpts gave a very human perspective on a story that otherwise has been mostly about the numbers.
In adapting the profiles, we also tried to include some of that perspective in the notifications.
The Notifications
This experiment differed from earlier jobs report experiments in a few ways. We tested a new format, as well as new timing.
First, because Jana’s piece came out on the Saturday after the jobs report was released, we needed to alert our jobs report subscribers that we were going to be changing the timing of our experimental alerts, which we had been sending to coincide with the release of the numbers early in the morning of the first Friday of each month.
We sent our first notification on Thursday afternoon, and told our jobs report audience that we would not be sending our experimental jobs report notifications at the normal time on Friday morning but to look for us on Monday.
Then on Monday around 2pm ET, as promised, we sent a series of sequential notifications, each linking to a specific profile within Jana’s article.
We called these notifications snapshots, because each of the five contained a short quote from one of the people interviewed in Jana’s piece, and included their photo as the notification icon. The title of each notification included their name and age.
The action buttons in each snapshot offered users the option to go to the full story, which, with the assistance of the Guardian US interactive team, deep-linked to the area of the story that had that person’s profile. The snapshots were also persistent, meaning that even if you tapped on the notification itself, rather than the buttons, the notification stayed in place until you dismissed it.
We followed up these snapshot alerts with a final notification, sent at 5pm ET on Monday, with a call-out to users inviting them to tell us their stories.
This built upon the call out from the previous month, when we had first tried a call-out through notificationsm which hadn’t elicited any replies:
In response, and to further explore whether notifications can be effectively used for crowdsourcing, we made a few changes to call-out format for this month.
- First, we asked users a specific question, and made clear that their answers might be used in a future story.
- We also sent the alert much later in the day. The previous month’s alert went out at 9am ET on a Friday, where this alert went out at 5pm ET on a Monday.
- We also changed the method through which users submitted responses. While in the earlier call-out users responded through a Google form, this time we lead them to a submission page on The Guardian US site, which had the added benefit that we were able to more prominently display our branding.
Last, on Tuesday, we sent a notification with a survey asking for feedback on the experiment.
How They Worked
With the different types of notifications sent, there are a few interesting metrics to consider for this month’s experiment:
We sent the snapshot notifications to 303 subscribers and registered that they were shown to 49% of users. 63% of users closed the first notification, and 25% tapped on the first notification itself to open the article.
In addition, 8% of users tapped on the action button of the first notification to begin the snapshot sequence. Of that 8%, 75% made it all the way through the five notifications, and 42% then clicked through to Jana’s full interactive.
Our call-out notification was shown to 52% of subscribers, of which 47% closed the notification and 17% tapped on the notification itself, which brought them to the submission page. An additional 6% tapped on the “Tell Us” action button, which had the same effect.
In an exciting development, we received nine responses to our call-out through the submission page, which was really encouraging after the lack of responses the previous month. These users told us how the last eight years had affected them, and who they would be voting for for President — both remarkably personal things to share — and told us we could contact them.
We believe this demonstrates that they both understood and enjoyed the experience enough to want to share, and that call-outs through a notification, when properly framed, can be an effective way to ask audiences for participation.
What We Learned
As always, the survey was a valuable tool for gauging the effectiveness of the experiment. Our 21 survey respondents told us more about how the call-outs and profile-driven notifications worked, and continued to shed light on our jobs report audience, who we’d been experimenting alongside for six months.
The snapshot notifications conveyed a more personal experience. It was the quotes, our survey respondents told us, that made the notifications feel more personal, and to a lesser extent the head shots as well, rather than the prompt we used to move through the notification, “Meet ______”, or our call for their personal stories.
This type of format might also be a way to connect readers with a topic they don’t have a particular affinity for — by using personal narrative. One user wrote in to tell us that they thought you didn’t need to be interested in economic news to be interested in these notifications.
Timing, or some still unknown element, still dictates when users choose to interact. As we continue to explore this topic with a dedicated user base, we’ve also started to ask questions about the users themselves, and that get to what makes them engage. This time, we asked users, “If you didn’t tap through, why not?”
Some users told us they didn’t receive the alert, which we know to be an ongoing problem. Others said they just weren’t interested. However, 27% said that it was because they received the notifications at an inconvenient time. We sent the snapshots at 2pm ET on a Monday, hoping to hit a midday audience looking for a break, after seeing very little engagement on sending in the early morning the previous month.
This month’s 8% tap-through rate is slightly lower than the 10% we saw in for both the September jobs report experiment (for which we also sent sequential notifications) and the August experiment. In September, we also sent the alerts later in the day, while in August we sent alerts at 10am ET on a Friday.
An interesting observation is that none of our users chose “The icon threw me off” as a reason they didn’t tap through. For this month’s experiment, our icons were a penny, and then head shots of the interviewees from the article. None of our icons had Guardian Mobile Lab branding, yet, this didn’t seem to confuse the audience.
While we could have been more specific in the way we worded the response (current wording doesn’t allow for the possibility that a user doesn’t know what an icon is), it’s rare that zero people select one of our multiple choice answers, so it’s worth theorizing why that might be.
For this experiment, we delivered non-Guardian icons because we were trying to challenge the notion that an alert icon needed to be used for news source branding or attribution. We wanted to explore the possibility of using that space (since space is so limited in an alert) to convey different signals or supplement alert content.
Based on the survey response, there are a few possible interpretations. First, as mentioned above, it’s possible users didn’t know what the term “icon” meant. Second, it’s also possible that users don’t pay as much attention to alert icons as we think they do, or third, users understood that the alert was coming from the mobile lab without needing it to have a branded icon. With further testing, we’ll hope to be able to confirm whether or not publishers can start reclaiming the icon space for something other than branding.
Users either appreciated or didn’t notice that the notification was persistent. Implementing the persistent notification (it didn’t disappear when it was tapped on) was a good on-the-fly suggestion from Connor Jennings, one of the lab’s developers, and the majority of users either didn’t mind it or found it convenient. Because each individual notification was unique and deep linked to the relevant area of the interactive, this also meant the user could use this additional navigation at their convenience rather than having to choose which notification to tap through on and losing the rest of the experience.
Users weren’t sure what to think of the call-out. The survey responses to the call-out were a bit of a mixed bag. While 29% of users said they liked it, the same number said they weren’t sure about it, and the rest either didn’t like it or weren’t sure what we wanted from them.
So despite the responses we did get, we think there is an inherent challenge to making call-outs through notification effective, because of the lack of in-line replies, which are are not currently available through web notifications.
While we were excited about the increase in replies this month, there’s still clearly more to learn there both in terms of signaling what we are looking for, and how we are asking for it. In-line replies are still a ways off for web notifications — they’re an area we think is worth exploring but the technology needs to develop further before it becomes a seamless experience.
We’re starting get a sense of a good sequence length. This month we sent nine notifications, including the notice that the experiment would be different, the six-part sequence, the call-out, and the survey. While the engagement with the snapshots was slightly lower than that for sequential notifications in our previous series, the vast majority went all the way through, and drop offs were at the beginning of the series rather than atrophying toward the end. In addition, engagement with the call-out, which came later, was much higher than it had been for previous call-outs. Finally, survey respondents overwhelmingly told us the notification number was appropriate.
We’re taking a break from jobs report experiments for now while we focus on experimenting in other areas.
We always enjoy hearing from readers and those experimenting in the mobile space. Let us know if you have additional ideas and observations in the comments or email us: innovationlab@theguardian.com. We look forward to hearing your thoughts.
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
