The Pitfalls of Measuring ‘Most Popular’

At the BBC, we spend a lot of time trying to understand and measure how media is consumed across the many screens in our audiences’ lives, from the big screens in our living rooms to the small screens in our hands. As a major media provider that delivers live and on-demand content online, there are seemingly endless questions that can be asked about the evolving habits of the BBC’s digital audiences. How many platforms, on average, do people use? What are the affinities between BBC brands and genres? Does the public appreciate what we do, and how can we measure that?

Delivering and evaluating the Rio Olympics

Every now and then, a major event comes along — Glasto, royal weddings, Wimbledon, World Cups, a referendum — and the BBC mobilises its resources behind covering it in whatever ways that the audience might want to experience it, from traditional television and radio to webcasts, catch up, and packaged clips. We’ve a huge heritage of bringing the nation together for such events and increasingly, this is happening online.

One such event where the BBC delivered shared moments to the nation and the world was the 2016 summer Olympics in Brazil. As Ben Gallop, BBC Sport’s head of digital and radio, said after the Olympics, “With Team GB performing heroics, we wanted to deliver all the medal moments and breathtaking action to audiences wherever they were online.”
 So, once the 2016 Olympics medal count was tallied and Britain had unbelievably bested its performance in 2012, and Usain Bolt once again mesmerised us, and the Ryan Lochte farrago had blown over, the BBC went to work trying to make sense of it all. How did 2016 compare to 2012? Did people watch the Olympics live online or was it mostly catch-up? How interested were audiences in highlight clips? What can we learn for next time? And of course, what were the most popular events online?

The Andy Murray Mystery

As a pan-BBC team in practice and in spirit, the ‘most popular’ question fell to the Digital Analytics team. Of course, most popular is not a metric, but intuitively, the proxy for this could be the number of times that something was played. A simple sum of ‘stream starts’, as we call them, against unique BBC programme ids — or ‘pids’ — should do the trick. When we ran this data, we saw that the webcast of the gold medal match between Andy Murray and Juan Martin del Potro had the highest stream starts for any event across the entire Olympics. This was slightly surprising being that the match was also being broadcast live on BBC Four and later on BBC One, but it seemed plausible enough. We also looked into which events had the highest cumulative minutes watched as we thought this could be a better indicator than stream starts for judging audience engagement. And that’s where we saw that the Murray vs Del Potro webcast fell far short of the top. How could the event that was streamed more than anything else — a tennis match which lasted nearly four hours — not be near the top of the list for total minutes played?

Looking at the match a little more closely, we broke down the total playing time within one-minute increments throughout the match and found that there had been a huge spike around the 33rd minute and then a sharp decline shortly after. What had caused this mysterious swarm of traffic? Was it something that happened online, or something on television? We scoured social media timelines but nothing suggested that anything extraordinary or momentarily ‘viral’ had happened online. Finally, we turned to a recording of the live television broadcast of the match itself.

Watching the BBC Four broadcast at the exact moment when we saw the spike online, we spotted something: just as the match was heating up, BBC Four informed viewers — using an inconspicuous notification on the right-hand corner of the screen — that the broadcast would continue on BBC One. It then cut to the studio for an update of the day’s Olympics events so far. This is when people must have scrambled for an online stream of the match, only to arrive at a webcast of what they were probably already watching on television. Understandably, they left as quickly as they’d arrived, but not before registering a stream start in our analytics system. The accumulation of these fleeting stream starts elevated the Murray vs Del Potro match to the top of all Olympics events played on the BBC online.

The Murray spike highlighted a few things for us: first, don’t settle for the most obvious metric. Even if the data seems plausible, ask follow-up questions. Second, the answer is often not in the data itself, but in a bit of context that needs revealing. Finally, and more broadly: people have expectations around what’s available to watch. If it’s live on television, it should be live online. The second or third screen is never far.