Streamroot Data Science Series #1: per-Browser QoS Analysis

We’ve all been there: that crucial moment in our favorite series where all of a sudden, video froze. Video quality is a topic close to everyone’s hearts, binge watchers and broadcasters alike.

In addition to the amount of traffic offloaded from CDNs by our distributed delivery network (Streamroot DNA), quality of service (QoS) related metrics are crucial for our customers. This is why our Data Science team religiously monitors and analyses our clients’ QoS metrics. Our 20 million daily video sessions give us a unique understanding of QoS, and in this first article of a series, we would like to share with you some of our insights.

As most of us are in a quest for the best QoS to watch our favorite series, many viewers wonder what is the best browser for watching video.

To answer this question, we took a month’s worth of data with a 1-hour granularity, keeping only the multi-bitrate streams, i.e. videos that have more than one quality available, which changes in real time according to viewers’ internet connection to keep smooth playback. Streamroot is compatible with all modern web browsers, and we picked four of the most common ones across our customers’ audiences: Chrome, Firefox, Opera and Safari. For each browser, we focused on the most common major versions.

From this dataset, we plotted the average values per browser and per major release of four of the key QoS metrics we constantly monitor:

• Buffering Ratio — measures the time a player spends buffering as a ratio of the total play time. We’ve all experienced buffering before: when your video freezes and this annoying loading wheel appears. Even though it depends a lot on the use case, we can say that up to 1% buffering ratio is very good, anything under 0.5% is excellent.
• Track Switch Events — the number of times video quality switches up or down. The acceptable number of track switches would be the same as the number of available qualities, usually no more than 6 per session.
• Average Bitrate — usually in Megabits per second (Mbps), but you’re probably more familiar with other terms for video qualities like HD, 1080p or 720p. Basically, the higher the quality, the higher the bitrate. A medium quality video streams at around 1 Mbps while an HD stream would be 6 Mbps.
• Dropped Frames Events — a video is a collection of ordered frames and it can happen that some of those frames are not displayed, resulting in a jagged rendering of the video. This phenomenon starts to get visible above 60 dropped frames per minute, it becomes annoying above 150 and is barely watchable over 300.

No winner takes it all

Looking at the data, first thing we can see is that no one browser performs better than the rest on all four metrics:

In Bitrate, Opera takes the lead, closely followed by Safari, Chrome and Firefox.

When looking at the Buffering Ratio, one of the most important metrics in streaming, Safari leads ahead of the competition with an average close to 0.5%, followed by Opera, Chrome, and Firefox.

Moving on to Track Switch Events — Safari takes first place with an average of 4 track switch events per session, followed by the very close second, Opera, Chrome comes third, while Firefox closes the list once again.

However, track switch events do not make much sense without also looking at bitrate and vice versa. More track switch events are not necessarily bad if they come with a higher average bitrate: it means viewers switch tracks, but they switch for a higher quality on average. An even better situation is when track switch events go down and bitrate goes up. It means viewers are more likely to have a high video quality and stay on that quality for longer periods of time. That’s what you can see above for Opera releases.

Our fourth metric is Dropped Frame Events. Here, Firefox and Chrome lead with Opera not far behind. Safari, which demonstrated good performance so far, surprisingly comes last with 95 dropped frame events per minute, almost 4 times more than Opera! One could stop here and think that Safari is underperforming. A data scientist, however, would be intrigued by this anomaly — this does seem like abnormal behavior… how can we explain this one? Does Safari really drop that many frames, causing a poor viewing experience?

Digging into distributions

Let’s zoom in on dropped frame events and see how each population (Chrome 64, Chrome 65, Chrome 66, Firefox 52…) is distributed for this metric.

First, a general comment: you can see that most populations stick to the left of the distribution, that is, to the low dropped frame events per minute. This is where you find all Firefox versions - more than 60% of their populations are located in the 0–15 dropped frame range. Same for Chrome 64 and 65.

Let’s get back to Safari’s abnormally high average dropped frame events per minute. If you compare Safari’s distribution to other browsers you can tell there is something wrong: it spreads from the first range up to the +150 dropped frame range and not one range reaches more than 30% of the population.

This distribution actually illustrates the Safari “open tab” issue. A Safari user “virtually” experiences dropped frames whenever switching tabs and the longer she spends on another tab, the higher the dropped frame rate becomes. This is because Safari considers that a frame is dropped as long as it is not visible. As we can’t tell how many people switch tabs and for how long, we can only intelligently assume that this is the reason we see this highly spread distribution going up to crazy values, above 150 dropped frame events per minute, causing an extremely high average value.

Know your Biases

One important thing to note regarding this analysis is the possible effects of confounding factor and bias. When we want to explain the effect of X (browsers) on Y (QoS), there could be another extra variable Z that affects both X and Y. In our case, Z, the confounding factor, could be, for example, the quality of the user’s hardware. For example, Safari users are likely to have more powerful computers, which might account for their excellent QoS more than the use of Safari as their browser.

Let’s actually look into that. We split our dataset according to three operating systems: Mac OS X 10, Windows and Other, which includes operating systems such as Ubuntu and Android. We scatter the resulting series of points in the graph below. The x-axis represents the number of track switch events per session; the y-axis shows the buffering ratio. The size of each point is proportional to the video bitrate.

This graph tells us that Mac OS X 10 users have better QoS in general and confirms what we suspected. Considering that the majority of Mac users don’t use Safari as their browser, we can reasonably conclude that there is a bias due to the fact that Safari viewers use a Mac. Our comparison at the beginning of this article did not stand because Safari users had an advantage compared to other browsers. To make it fair we need to eliminate this advantage — by testing all browsers on Mac. To do that we plotted the same 3D representation as the one above, but this time we only kept Mac OS X 10 users and split by browser:

As you can see, when leveling the playing field, Safari is not the top performer: most of the Safari points are located between 2 and 6 track switches per session and between 0 and 1% buffering ratio, and the point size is between 2 and 4 Mbps. On the other hand, Opera spreads from 0 to 7 track switch events per session and sticks to the 0 buffering ratio line and the size of its points demonstrates a good bitrate as well.

Conclusion

In this article, our aim was to demonstrate how a simple question as “what is the best browser for watching video?”, is actually not that simple at all. Manipulating data can be tricky, and it’s very easy to jump to false conclusions. While at first glance we saw that Safari seemed to be the top performer, digging deeper allowed us to discover that it is not the case when isolating the influence of the Mac operating system and hardware.

This is only a glimpse of how we analyse and use our data at Streamroot. While this exercise provides a simple example, this is something our data scientists encounter on a daily basis. Whenever we analyse data we always keep in mind the biases that could affect our results, such as the hardware, type of stream — Live or VoD, the content duration or the countries in which viewers are located, to name a few. Having a dataset is good, knowing your dataset is better.

Stay tuned for our next articles in our QoS series, where we’ll share with you some more insights into interesting statistical methods that provide more information than simple averages: variance, Chi² analysis and more.

And if you’re as passionate as we are about data and video QoS metrics — check out our Careers Page or send your CV to jobs@streamroot.io — we’d love to have you on our team.