Crowdsourcing Through Notifications: Lessons Learned From the Jobs Report, Round 5
In which we see if users want to talk back.
After we ended the Good News/ Bad News paradigm in August, we went looking for a new way to frame the next installment of our ongoing jobs report notifications experiments.
One of our main takeaways from the Good News/Bad News series was that our assumptions about the audience’s need for information had been off. That audience, which largely signed up via the Guardian’s business live blog, had already indicated through one of our follow-up surveys that they were a business savvy crowd. They already knew what the jobs report was and when it was released. Our experimental notifications (which have always been separate from the main app notifications sent from the news desk), had evolved to become a kind of explainer, and did not meet our users’ need of wanting the numbers — and fast.
So with this in mind, we designed a new idea for the release of the August jobs report in early September. Business reporter Jana Kasperkevic had told us that she had the opportunity to interview Tom Perez, the US Secretary of Labor, on the afternoon following the jobs report. The labor secretary typically makes some time available for reporters on jobs report day and Jana was willing to devote some of her time with him to posing questions from users.
We decided to run with it — we had talked within the lab about wanting to try out ways of giving some of our notifications audiences the opportunity to respond — and planned a call-out to users through a notification, with the idea that members of this business-savvy audience might have specific questions they wanted to ask Secretary Perez.
There were inherent limitations, though. In-line replies are not a feature of Chrome web notifications, though they may be in the future. (A reminder: we are still running our notifications experiments only on Chrome on Android devices, though we are actively planning and building capacity for iOS experimentation). Because of this, our notification would link to a form where users could submit questions. We chose Google Forms because they are easy to use, work well on mobile, and we had experience asking our users to head to a form from a notification in order to answer the surveys we send after every experiment.
In terms of satisfying our audience, we also wanted to send our notification closer to the actual time of the jobs report at 8:30 am ET, and the requirement of lining up all the notifications prior to sending in the earlier Good News/Bad News experiments had slowed us down. Because we only needed the one top line “jobs created” number from Jana, we were able to send our initial alert to our subscribers at 8:37 am ET.
Our first alert gave the top line job number and also a teaser to the experiment we would run. It also told users we would be crowdsourcing questions for the interview later that day. The action buttons gave users the option of submitting a question or going to the article, and tapping on the notification also led to the Google submission form. We also told users we would send around another chance to ask questions.
About a half-hour later we sent the second notification. We gave users until 11:30 am ET to send in their questions, and told them that we would send along a selection of answers after the interview.
Analytics showed that at least a few users considered submitting a question. Overall, 15% of users, between the first and second notification, tapped on either the action button or the notification itself, which both led to the Google form.
Unfortunately, we didn’t receive any questions. Some hypotheses we’ve considered for why this might be the case:
- Timing: We still have a lot to discover as it relates to the best times to send different types of notifications. We’ve seen deeper engagement and better response rates to alerts sent in the late afternoon 4–5:30 pm, so the morning might not have been the best time to ask someone to fill out a form, since they’re busy with work/life/email/etc.
- Name recognition: It’s possible that our users, as business savvy as we believe them to be, might not know Tom Perez by name and so be less inclined to ask him a question.
- Self-interest: We think it may be possible that people are more apt to ask questions when the topic is personally relevant to them. Asking the secretary of labor a question may not have been clearly relevant enough.
- Ease of use: Another possibility is that we were just asking people to jump through too many hoops. They may have been more inclined to respond if it were an easier experience, such as via an in-line reply (like a text).
- They didn’t see it: People may have stopped reading the alert after they got the jobs report news they wanted at the top of the alert, and they didn’t read the “STAY TUNED” portion.
Though we didn’t receive any responses later in the afternoon we worked with Jana to take the questions she had asked Secretary Perez and adapt them into three notifications.
Jana’s interview with the Labor Secretary was scheduled for 2 pm ET. After she finished, she quickly transcribed the conversation for us and we wrote an opener and three sequential notifications, including questions and their answers, based on her copy. We sent those at 4 pm ET.
In our notifications we tried to signal the topic of the question in the title, and included both the question and answer in the body of the notification. We attempted to put the question in the title, but found them too difficult to summarize in fewer than 27 characters, which is the limit. Instead, we used the title to try to frame the topic of the notification, and put both the question and answer in the body copy.
Results by the numbers:
- 314 users were sent the first of the sequential notifications, which were shown to 162 of them.
- 41% dismissed the notification
- 10% tapped directly on the notification, which in both cases ended the sequence.
- 9.8% of users tapped the next button to see the first of the questions and answers.
- Of those that began the sequence, 75% finished it.
- 12% of those that finished the sequence tapped the Open Article button on our final notification to open Jana’s article which featured more news about the jobs report and further quotes with from the labor secretary.
Our Q&A with Tom Perez was interesting to write for notification but ultimately didn’t grab readers. That’s ok. We’re looking for new ways to continue to experiment with the jobs report and would definitely to test crowdsourcing through notifications again, either with changes made to the process, or on another topic.
Have you tested crowdsourcing through notification? Let us know what you think. Send us an email at innovationlab@theguardian.com.
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
