What We Learned Covering the Olympics Through Notifications
The Olympics! The wonderful world of sports! The mobile lab was excited to use the 2016 Rio Olympics as our latest testing ground for our experimental web notifications for Android.
Over the course of the games, from August 5 to 21, we ran an experiment exploring new types of web notifications and testing user behavior around settings preferences. In it, we tested four formats: real-time medal alerts, a daily team leaderboard, quizzes, and a real-time poll we called a “morale meter.”
The Olympics presented a few new and distinct opportunities to add to our bench of notifications experiments. First, the Rio games, lasting two and a half weeks, were far longer than the day or two of earlier events, like the US presidential primaries and the EU referendum, around which we’d run experiments. The increased time frame allowed us the chance to see how people responded to web notifications for more than a few days, and to take a close look at their behavior around managing their settings and interacting with a fairly complex sign-up page.
We wanted to see how users reacted, study their level of engagement relative to shorter events, and see if there were prescriptions on format and timing to be gleaned from their response. We also tested a new notification type: quizzes, taken fully in the notification itself.
Below is a detailed account of the experiment, followed by overall findings.
How we recruited the audience:
We recruited users for these notifications using methods that had worked well for earlier experiments, but also reached out to our audience in a few new ways.
As in previous experiments, we created a sign-up page. Intended to be viewed on a Chrome browser on mobile (though functional on desktop as well) users were able sign up for the Olympic teams they wanted to follow as well as what we took to calling the “big bucket,” which was comprised of the three other types of notifications we offered: the morale meter, a daily leaderboard and quizzes.
We also recruited users by placing a link to a description of our experiment in the top of The Guardian’s opening ceremony live blog. The link led to a Medium post, we had written explaining the experiment and tweeted it from @gdnmobilelab.
We also tried recruiting in some new ways. First, we adapted our Medium post into a short article for the Guardian website describing what users could expect, how to sign up, and what the technological requirements were for the experiment (for example, the alerts were only available on Chrome). Then, working with the Guardian US audience team here in New York, we posted our Guardian article to the branded Guardian US and Guardian Facebook pages, as well as the Guardian US and Guardian US Twitter accounts.
Additionally, we also created an article embed of a shortened version of our sign-up page, from which the user was able to sign up for the experiment directly from an article. While this wasn’t available until over a week into the experiment, we eventually included the embed in two articles: ours, announcing the experiment, and a separate article produced by the Sports desk. We gave users the option to sign up for the big bucket as well as for medal alerts for the three teams — the US, UK, and Australia — the three teams with the greatest number of sign-ups at that point, and that correspond with the Guardian’s core audience.
We made the embed a shortened version of the full sign-up page in order to avoid introducing competing scrolling, between the article itself and the embed, in the mobile view. Between the two articles on which it was placed, 1,426 users saw the embed.
Aside from adding the embed, we made one substantive change to sign-up during the course of the experiment. At first, we limited the number of medal alert teams a users could sign up for to three, hoping that this would encourage users to come back and manage their settings when there was an important event or story thread that they learned about and wanted to follow, that correlated with a different country.
After the first few days though, we realized the limitation wasn’t necessary, and more crucially, it was keeping users from adding teams. We removed the three-team limit to offer all 207, and found that this offered a new and better set of data, and greater insight into personalization preferences of users.
Some highlights of our recruitment:
- Overall we got 2,586 unique subscribers, a number which grew incrementally over the course of the two weeks
- There were 11,580 unique page views of our sign-up page
- At least 6% of subscribers, or 756 people, revisited the sign-up page and changed the notifications they had signed up for at least once
- 32% of subscribers, or 828 people, signed up to receive real-time medal alerts for at least one country, and also signed up for the big bucket notifications
- 48.7% of our users, or 1,259 people, signed up to receive real-time medal alerts from at least one country, but did not sign up for the big bucket that included the quiz, leaderboard and morale meter notifications
- 19.3% of users, or 499 people, signed up to receive the notifications from the big bucket only.
After the end of the experiment we sent out a survey which was completed by 9.2% of all subscribers, or 239 people. In addition to data, the answers gave us qualitative insights into where our users came from.
- 44.4% of the survey respondents said they found out about Olympics experiment through an Olympics-related Guardian live blog
- Another 24.7% said they found out about the notifications experiment from an article page that had our embed on it
- 59% told us they were loyal Guardian readers, with another 28.9% describing themselves as casual Guardian readers
- 85% told us they were also following the Olympics on television, 63.5% followed through Guardian live blogs, 62% said through other news sources, and 24.9% said on Twitter.
Olympics Notifications Formats
Daily leaderboard alerts:
For every full day of competition at the Olympics, we sent out a daily leaderboard of the day’s the medal count. The idea for the daily leaderboard was based on the auto-updating format we’ve used in previous experiments to show vote results as they come in.
In this version, instead of vote counts and percentages, we notified about the top three teams on the Olympics leaderboard, and the number of gold, silver, and bronze medals they each had.
Our leaderboard notifications arrived on the lockscreen around 11:30pm ET, after the full day of events had ended. The events went quite late, and so we thought this was the best option to reach as much as of our audience, as international as they are, at a time when the our daily results would be reliable and complete. In deference to the fact that 11:30pm ET is after many people’s bedtimes, and before many in the UK wake up, we made sure that our notification arrived silently.
Users who tapped on the notification itself were sent to the live blog, while users who tapped on one of the action buttons were able to manage their settings, or were sent to were sent to the Guardian UK visuals team’s full interactive page with in-depth statistics on each team’s performance.
A persistent consideration with auto-updating notifications is sound. In earlier experiments, we have made sure that each auto-update was silent by tying it to an earlier notification, like a subscription confirmation, where sound might be expected. Recently released versions of Chrome allow for all notifications to be silent, but because our daily leaderboard updates went out in the late evening, we wanted to be sure that these would arrive silently, too. We double-proofed them by continuing to auto-update off of the earlier alert to make sure that those who had not updated their browsers would not get a noise-making notification.
We found that the leaderboard notifications were a reasonable driver of traffic to the live blog, with 21% of users tapping through at one point or another. In the survey, 85% of our users said they would sign up for them again, and 78.4% said they were useful or very useful.
While those numbers are a strong indication of the value of the format, we also received some useful feedback from users on our survey who said they would have appreciated a bit more editorial context in the leaderboard alerts. In addition to the table, they suggested we add a line describing what had changed since the last alert, for example, “After today’s competition, China has overtaken the US on the leaderboard.”
In our planning for this experiment, we thought about doing this but eventually decided against it because we would have needed to add the text manually, increasing the workload for our small team. Just adding that one line of text would have changed the notification type from auto-updating, which is data-driven and thus automatic. With a larger team, or a team working in shifts, this is something that we would like to explore doing in future experiments, and we definitely see the value of the idea.
Real-time medal alerts:
The Olympics is a dynamic event. In addition to finding out all about sports you never knew existed, like canoe slalom, there’s a thrill to seeing your favorite teams and athletes win medals in competition against all the world’s best competitors. We wanted to capture that dynamism with real-time medal alerts which we constructed using the data feed to which the Guardian subscribes.
The Data Feed: Our real-time medal alerts, as well as our leaderboard, were built off of the work done by the Guardian’s visuals team in London, specifically by Fielding Cage and Will Franklin. The Guardian, in coordination with several other media outlets, had access to Olympics data through an API administered by the Press Association, off of a feed provided by the International Olympic Committee.
We queried feeds set up by Fielding and Will for the data that populated our notifications. For example, in our real-time medal alerts, Will provided a feed that listed all the teams and within those, all the sports and winners. To get the data for the alerts, we hit the feed every 10 seconds to check for a difference, meaning a new medal had been won, and sent that out to those who had subscribed to the corresponding team.
For every country for which the user was signed up, they got a notification each time a gold, silver or bronze medal was won. As described earlier, we originally decided to cap the number of teams a user could sign up for at three. This was done as a way to see if we could get users to manage subscriptions more actively. When we removed the cap we found that users were more apt to manage their subscriptions when they could add, rather than swap teams.
The majority of users (1,638) or 64%, subscribed to only one team’s medal alert, followed by 22% who subscribed to two teams (577), 6.8% to three teams (177) , and 5% to four teams (131). But we were interested to learn, there were users who subscribed to more than 30 teams, and actually there were two users who subscribed to all 207 teams. While there is a point at which most users will self-cap, some users clearly liked having the freedom to decide their settings.
In our survey, 85.7% of our users told us they found our real-time medal alerts useful or very useful, and 84% said they would sign up for them again. The notifications, which when tapped took users to the live blog, had a decent 15-20% click through rate. We received feedback from users in our survey that they would have preferred to have been sent to a piece of specifically related content — an article or blog post about the event. While we can understand that desire, meeting it wouldn’t have been possible because of the heavy workload (we would have had to send each individually) and because we didn’t have content for every medal won.
We also received very useful feedback that we could have done a better job in phrasing the alerts so that the compacted view might show the sport, rather than repeating the country. We think this was a very good point, and something we will think about in the future.
Quizzes:
During the Olympics experiment, we also debuted quizzes that could be taken wholly in a series of notifications. We sent three quizzes: two created by shortening two quizzes written for The Guardian by Nancy Frostick and sports writer Paul Campbell, and one created with the Guardian’s interactive team in the US about the marathon, in conjunction with the RioRun interactive app they developed.
Building off of experiments we had done with interactive and sequential notifications earlier, the quizzes were again designed to test if users could navigate through a series of notifications using the action buttons. Tapping on the alert would bring the user to the full version of the quiz in a mobile browser, while using the action buttons would lead the user through a shortened three- to five-question version.
At the end the user would see tabulated results with copy that provided context about how they’d done. For example, those with five of five questions right were told they got a gold medal, those those with four of five correct earned silver and so on.
We also included a small Twitter integration. One of the action buttons included in the results notification invited the user to tweet their score. Tapping the button opened Twitter in a mobile browser, with pre-populated text announcing the score. We had one user who did this, proving that the functionality worked!
Editorially, the quiz format and the notification character count required us to pare down the original quiz questions written for and published within a standard article format. Because there are only two action buttons, we needed to remove two answer possibilities from each question we adopted. We also needed to shorten some questions, or recognize that they were too complicated to send, in order to get each question to make sense within the 240 character limit. Likewise, answer choices needed to fit into the space allotted to action buttons. We also randomized the buttons in order to scramble the answers, and to test the send/show rate again, as this has proved problematic in the past.
We sent each quiz at a different time of the day, aiming to gauge whether the variation changed the response rate.
The quizzes, as part of the big bucket, were sent to 1,327 people, around half of our total user base. Still, we believe we may still have lingering issues with shows on notifications with randomized buttons.
Our first quiz, sent on Wednesday, August 10 at 12:45pm ET, registered 347 shows of 635 notifications sent. Of those that received the notification, 53% closed the notification, and 20% tapped immediately on it, sending the user to the mobile version of the quiz and ending the sequence. 7% of users engaged, and began the notification series, and 20% of that finished the quiz to see their results.
The engagement rates were similar for our second quiz, sent Saturday, August 13 at 12pm ET, though our subscriber group size increased. We sent our three-question marathon quiz to 986 subscribers, 570 were shown it. Of that 570, 59% of users dismissed the notification, and 12% tapped on it directly. This time, 9% of users engaged and started the quiz, and of those 63% finished it.
Our third quiz, a five-question recap of the games, was sent on Friday, August 19, at 5:11pm ET to 1,297 users. We registered it showing to 642 users of which, 46.7% dismissed it, and 12.4% tapped directly on the notification itself, leading to the full quiz in a mobile browser. 6.6% of users tapped on the action button to start the quiz, and of those that started, 60% finished it.
Despite varied timing, engagement rates remained relatively static throughout the quizzes. Further questions about the quizzes in the survey revealed mixed reactions. Of those in the survey that said they had received the notification, 43% said they found the quizzes were fun and easy. 62.4% of users rated them not interesting, in contrast to the majority who felt the opposite way about the leaderboard and real-time medal notifications. 44% of our survey respondents said they would not sign up for them again.
While technical challenges continue, we learned editorially that quizzes adapted from content were relatively easy to produce. We’ll continue to refine the format to make these more intuitive and interesting, but it may be because they do not fulfill an urgent or time-sensitive need that quizzes are just of less of value for many users in an alert format.
Morale meter:
Last, we adapted the live polling notifications we used for the EU referendum for morale meter notifications sent leading up to a major Olympic event.
Inspired by a screen I saw at Back Bay station in Boston, morale meters were intended to take the temperature of our users’ sentiments about an Olympic moment or major competition, and also alert potential viewers and readers to the imminent event itself. Like quizzes and the daily medal leaderboard, morale meter notifications were part of the big bucket of notifications we sent to at least 725 subscribers.
We sent two morale meters over the course of the Olympics, and though the content differed, the format was the same for both. The first, sent on August 11 at 2:33pm ET, asked users if they thought Simone Biles was unbeatable as she went for the women’s gymnastic all around title. The users were prompted to vote by tapping on one of the action buttons, which said “yes” and “no.”
We told users in the notification that they had a set number of minutes to vote, after which we sent back a tabulated result showing the user what the rest of the audience’s opinion was.
The Biles notification was sent to 725 users and was shown to 469 users. Of that number, 25% dismissed the notification, and 7% tapped on it, which brought them to the live blog. Additionally, 12% of users engaged by tapping on one of the action buttons: 82% of those who voted thought that yes, she was unbeatable (and she did in fact win gold).
The second morale meter was sent later that day, at 9:07pm ET, just before the first match-up of Michael Phelps vs Ryan Lochte in the men’s 200m butterfly. We asked users who they thought would win, Lochte or Phelps, and gave them a set amount of time to reply. That poll, sent immediately before the race, had a much lower engagement rate. Our data showed far fewer closes or taps on the notification itself or on the action buttons than alerts shown. This may be due to problems with our analytics and tracking. But, though incomplete, the data we have tells us that only 4% of users tapped on one of the action buttons to vote for either Phelps or Lochte.
The survey responses to morale meters also confirmed that our subscribers found them relatively unengaging. In our survey, 56.2% of people rated them not interesting, or very not interesting, while 39% said they would not sign up for them again. While after the EU referendum users said they found our live results the most useful and rated, as now, the polls as lower in value, we saw comparatively much higher levels of engagement with the polls during the EU referendum coverage than we did during the Olympics.
A preliminary theory for why users found these notifications less engaging: those who responded to poll alerts on the EU referendum were reacting to a real life situation, to which the answer would not be quickly revealed. When we asked users then what issue mattered most then, we posted the question in a moment of geopolitical uncertainty.
In the Olympics experiments, we asked moments before events unfolded, and on a topic that wasn’t critically important to their lives. This isn’t to say that entertainment can’t be part of these notifications, but we need to do more testing to establish whether the difference in reaction per topic is the critical factor.
Overall Findings
Complications With Data: Automated alerts are great since they nearly run themselves off of a feed during an event, but there’s a big upfront investment of time to review all of the data fields, nail down the best format and deciding how to code defensively against real-time feed errors.
The data feed required a fair amount of cleaning, and we weren’t able to practice with live data, which meant we had a few hiccups. Connor and Alastair worked diligently with Will and Fielding of the Guardian’s visuals team in London to get the data feed set up.
Even once that was done, there was still much to decide editorially. We had to figure out how to differentiate, in particular, the medals for one event from another. When the event involved a pair of athletes or a small team, like a relay, we had to think about how we would format the winners’ names in the notification. How specific did we need to be? How many different types of canoeing and cycling are there? There was a complicated taxonomy of event names to consider in order to ensure that the notifications read well and made sense.
This proved to be both an editorial and technical lift, because every decision made editorially affected how the feed was further delineated and presented. For other news organizations looking to do real time medal alerts or similarly automated notifications off of complex data sets, leaving time to make these considerations ahead of time is important.
User Behavior: We learned that people could locate and manage their settings. One of our hopes for this longer experiment was to observe user behavior around managing settings through a notification. We found that over the duration of the experiment there were 319 total taps on the Manage Updates button on the leaderboard and real time medal alert notifications.
The taps came to a fairly normalized curve, with the highest number of instances towards the middle of the experiment.
Audience Recruitment Highs, and Lows: We found that our recruitment saw quite steady growth over the first week, with a long tail of declining recruitment during the second week of the Olympics.
We tried a functional embed on a few relevant articles to see if people wanted to sign up in that context. If users could sign up with one tap within a piece of content they were already viewing — would that be successful? We only placed the embed on two relevant articles, so we knew it wouldn’t drive significant numbers. We also didn’t come up with the idea until a week into the games. Much of our learning this time out was in how to build the embed, and what to focus on in shortening and simplifying our sign-up to fit within it.
Low Engagement on Sentiment-Driven Notifications: Both morale meter and quiz notifications saw comparatively low engagement relative to our data-focused formats. We experimented with sending them at various times of the afternoon and days of the week, which had little effect. All of the morale meter and quiz notifications were sent during the afternoon, so further testing could tell us if we were overly narrow in the scope of our timing as well.
Low Engagement on Polls: Users didn’t respond to polls on Olympics events with the same gusto as they did during the EU referendum. We have a few theories to explain the difference in engagement for the two events. First, it’s possible that the threshold of impact was not high enough. While in the EU referendum, we asked questions about an issue that was highly important to those users, the polls we sent during the Olympics were on more entertaining topics.
The topic, coupled with the fact that users would soon find out the results of an event even if they didn’t interact with the poll, may have contributed to the lower engagement. It’s possible that sending a poll at a time that was more removed from the event might have compelled users to share more, rather than trying to build in our notification so close to the actual event.
In future experiments, we could try further targeting polls to a subject a user has shown interest in. For example: If we had segmented our real-time alerts by sport rather than by team, we could have sent the Phelps vs. Lochte poll to only those who had signed up for swimming alerts.
Quiz Format Effective: Even though our quizzes had relatively low engagement, the UI worked — and someone tweeted! This was our debut of the quiz format and we were excited to see how users responded. Between 7% and 12% of users who received quizzes started them, and of those, from around 20% to around 60% finished them. Our shortest quiz, at three questions, had the greatest rate of finishers, as compared to five-question quizzes, so we will continue to experiment with length.
Offer Personalization, It Will be Used: As it turns out, if you give people the choice of 207 teams, someone really will sign up for 207 teams. Two people actually! But more to the point, this experiment in asking users to select exactly how many alerts they wanted pointed to some interesting findings. While the majority of users selected only one team to follow (the US was the most popular, followed by the UK and Australia), 22% of users signed up for two teams, and just over 5% of users signed up for three and four teams each. Had we left our initial three-team limit in place, we wouldn’t have gained insight into the most frequently chosen preferences.
Further, these personalized medal alerts were more popular than the big bucket sign-ups, which speaks to an interest in greater personalization. Users were both able to and interested in selecting exactly what they would get. For news organizations looking to move forward in building smarter notifications, looking beyond the text of the notification, to the level of specialization offered will be an important step forward.
Relationship of Format to Editorial Choices: We’ve found that a notification format affects how we present the information, and vice versa. In running this and previous experiments, we’ve found a tension between writing for voice, and writing for utility. With some of our notifications, like the leaderboard and real-time medal alerts, utility was the primary function.
In order to have these notifications go out as quickly as possible, we had to make judgements on how to streamline the language so the notifications could be automated and go out seamlessly, even when adding additional information, like a change in leader or notice of an upset, might have also added value. With notifications like the quizzes and morale meter, we were able to inject more of a descriptive tone. Those notifications were neither urgent nor in as restrictive a format, so we had more time and flexibility to add voice.
As news organizations continue to move away from programmed headline notifications to those that are more descriptive and contextually driven, and in a variety of formats, they should take care that they do not lose the utility, whether the primary goal is for speed, explanation, or fun.
Interested in testing interactive notifications within your organization? Have questions? Send us an email at innovationlab@theguardian.com
The Guardian Mobile Innovation Lab operates with the generous support of the John S. and James L. Knight Foundation.
