Live Updating Notifications: What We Learned From Reusing the Format for the Super Bowl
Users found alerts updated with scores and commentary useful — more so than a single alert at the end of the game.
After the success of our live updating election notifications last fall, we were eager to find a way to test this type of data-filled notification again for a different type of news event. So we thought, what about the Super Bowl?
Why We Did It
In addition to wanting to follow-up on the election notifications, we saw the Super Bowl as another large planned news event that provided a good opportunity to test the formats on a lighter topic. We have wonderful sports editors here in the US who have live blogged the game for the past few years. (For readers outside the U.S., the Super Bowl is the final game in the National Football League’s annual calendar and a huge event for many Americans.) We thought that a live data alert sent from the Guardian might pique users’ interest. We also had collaborated well with the US sports desk previously for the Olympics, and for some of the lab’s earliest experiments.
A notification that combined data and commentary would, we thought, provide an informative and accessible way to get live information, including scores and plays. Further, we thought the live commentary through notification would be a way to differentiate our alerts from news alerts, which would likely only cover kickoff and the outcome of the game, as well as alerts from sports apps, which were likely to be even more data driven.
Finally, building this notification experience was a way to test and fulfill user interest in more of these kind of alerts. Following the elections notifications, 46% percent of users had told us that they might be interested in live updating notifications around sports.
Our Hypothesis
We believed that the appetite for live data we had identified through the elections alerts would be persistent in other kinds of events, and would resonate especially with a sports-focused audience. We thought we would see this in how users rated our alerts as useful or interesting, and in how they close, expand, and tap through on the alerts.
What We Did
For the Super Bowl, we worked with the UK apps teams to build a live updating alert into the Guardian iOS and Android apps that provided live scores and commentary during the game, updated by the Guardian US sports desk, with whom we developed the idea for what we would offer. The notification was designed to provide users with a way to stay up to date by glancing at their phone, while having the option to tap through for more information, much the same as in the election notifications.
We promoted the alert both in the US and globally through a sign-up screen that was shown to both iOS and Android Guardian app users in the US, as well as Android users globally, to notify them about the availability of the notifications and specifically target those who had previously read a Guardian story about football. From the screen, users could tap to sign up for the notifications. Android users were also able to sign up through their app settings.
We also worked with the sports desk including sports editors Tom Lutz and reporter Bryan Graham, as well as Sam Morris, a designer, to develop a live scoreboard graphic that would be included in the expanded notification.
Unlike in the election notifications, which also featured an automatically updating graphic with electoral vote totals in the expanded view, populated by a feed of AP data, the Super Bowl scorecard was updated manually by Bryan during the game. He was able to change the score and write in his commentary easily throughout the game using a tool created by lab developer Connor Jennings. In the expanded view of the notification, we offered a button to tap through for the live blog, as well as a Stop Updates button, which ended the notifications and unsubscribed the user.
We sent notifications during the first half of the game, took a break for half time, and then resumed for the second half. Our final notification was sent immediately after the Patriots scored their final touchdown during the very surprising and historic overtime. We sent a notification at noon ET the next day with a survey link asking users for feedback on the experience, as we do after every experiment.
Numbers in Brief
Audience Size: 6,923 users total
- 4,126 on iOS
- 2,797 on Android
20 alerts sent over the course of the game
Engagement:
- 10% of users who saw the sign-up screen opted-in to the alerts
- Users expanded the alerts an average of 10 times each over the course of the game
- On Android, one third of users tapped through to the live blog around twice. (Unfortunately, iOS data was unavailable.)
From the survey:
- Users who said the commentary was useful, easy to read, or a good complement to live score information: 91%
- Users told us it was clear or somewhat clear that the alert updated : 90%
- Users who said they felt they were kept up to date or very up to date by the alerts: 79%
- Users who said they would definitely or maybe sign up for these alerts again: 94%
The Data Told Us
The conversion rate from sign-up screens to alerts was about 10%. The total number of people who saw the sign-up screen was about 63,000, of which about 6,900 opted in, for a conversation rate of just over 10%. For contrast, the conversion rate for the US presidential election night live data notifications was higher, at 37%.
Some possible reasons for the disparity: First, the US presidential election was a global event that also commanded a larger audience overall at 230,000 users. Additionally, the Super Bowl sign-up screen was slightly different between iOS and Android versions, and also between the US and global audiences, which may have had an impact on the overall number of sign ups. In the future, we would want to make the sign-up experience more consistent for users on both platforms. As well as being good practice, it will also help us to get a better sense of the analytics across both platforms.
Users expanded the alert about 10 times during the course of the game. On average, users expanded about 10 of the 20 alert updates we sent. Of users who took our follow-up survey, 79% told us that they thought the notifications were a good way to stay up to date. One of the goals of our experiments has been to enable user choice. With that in mind, we didn’t expect to have a 100% expand rate on all alerts. We wanted to make sure the user felt informed and could get more information when they chose.
Users liked the combination of scores and live commentary in the same alert. In the survey, 78% of users who expanded the notification said they found the combination of scores and live commentary interesting or very interesting, and 79% said this combination was useful or very useful.
The majority of users found the scorecard helpful and easy to use, while about half of users said it gave them the information they wanted. In our survey, users gave varying reviews of the scoreboard: 66% said it was useful; 70% said it was easy to use; and 51% said it gave them the information they wanted. Only a few users — around 3% — said the alert was not useful to them.
In planning the scoreboard graphic we had two of Sam’s designs to choose from: one that showed which team had possession along with the score, and one that just had the score alone. We ultimately chose the latter: because Bryan was updating the score manually, indicating possession, which fluctuates more rapidly, would have introduced another layer of complexity that we weren’t sure we were staffed to handle. Additionally, this choice meant the Guardian US audience team, for whom Sam had also created the designs, could more easily use the scoreboard alone as Twitter cards.
In our survey, users said they might have liked that information about the team in possession (54%) or a visualization of the game statistics (41%), which we would like to work on for future iterations.
When users dismissed an alert they said they were doing so because they ‘got’ the information, not because they didn’t like it. We’re particularly interested in what users do with information in an alert when they don’t immediately tap through. When they clear or dismiss a notification, what are they saying? We asked users what it meant when they cleared a notification and were excited to see that the answer wasn’t that they were unhappy, but rather they had received and consumed the information provided.
We saw a very small opt-out rate of less than 10% on both platforms, while 77% of users told us that they knew how to opt-out. While we were unable to track when in the course of our experiment users opted out, most survey takers told us that if they did opt-out, it was because the game was over, not because our notifications had become burdensome.
Gut check: 95% of survey takers told us that these alerts were more useful than a single alert at the end of a Super Bowl. 94% of users said they would sign up for them again. That’s pretty good!
Additional Insights
Our general hypothesis — that a sports-focused audience would have an appetite for live updating alerts that include data and commentary — was proven. However there were fewer tap throughs to live content than we had seen in previous experiments. We had a few technical issues throughout which meant that users, particularly on iOS, did not receive all the notifications, which likely depressed the tap-through number.
The lower number of tap throughs may also be due to the more visual nature of the Super Bowl. Our users told us that on the whole they were also watching the game on television, which would have served as a primary means of information. We were offering brief analysis at a glance, but to further engage with that analysis would have required users to switch their attention to actively read about the game with slight delay, rather than watch it live.
Running this type of experiment required a fair amount of advance preparation work, as well as the work of an editorial staffer on the day. We started work with the apps teams in the UK on these notifications two months prior to the Super Bowl, to ensure their availability in the Guardian apps, a process that required a lot of focused communication. And because these alerts were manually sent during the game, there needed to be editorial resources attached to them during game time.
We were very happy to work with Bryan and think the notifications benefitted greatly from the amount of nuance and familiarity with the subject matter he was able to put into the live commentary. Without his contribution the experience notifications would not have been as conversational, informed, and quickly delivered.
We’re still thinking about the question: What is a complete experience in a notification? And another question we’re exploring further: what is a positive interaction with a notification and how do we better analyze that experience? It’s clear that notifications hold a lot of power, even in the small space in which they appear. Figuring out how to make a complete experience of them, though, is still clearly a very subject-specific task. For now, it is useful to have evidence that, for an intensely visual event such as the Super Bowl, users will have different needs and demonstrate different expected behavior than that of a less visual event, like the election. Watch this space for more thoughts.
