Why Netflix streaming works so well (hint: they cheat)

I’ve been asking myself why some services stream so well compared to others. Netflix is one of those that has always worked very well for me.

I wanted to know why, so I interviewed Jeremy Edberg, who was Netflix’s Lead Reliability Engineer for a while up until recently (he has his own company now). He gave me a really interesting answer.

The secret to serving static video content is to use CDNs. This is a well known fact, and any decent video streaming service will be using them.

If you clicked on this post you’re probably familiar with CDNs. There are quite a few different options for CDNs, and Netflix used Limelight, Level 3, and Akamai for a while. There is also CloudFront from Amazon and CloudFlare, just to name a few more.

But Netflix wanted the absolute best streaming they could get, while lowering cost. That’s why they created their own CDN which they named Open Connect.

This consists of machines that are highly optimized for delivering large files. That’s their main purpose, and they do an incredible job of it. In fact, they’re so optimized for delivering large files that Netflix still has to use Akamai for small assets.

In a nutshell, these physical machines have lots and lots of disk drives and are spread out all over the world in various tiers.

In one location, there may be a data center with a couple of racks that contain the entire Netflix library, and then there might be another data center somewhere else that only has 80% of the most popular content.

This is nice, because if the client is watching a video that is really popular, they will be directed towards the closest CDNs.

If they’re not watching something as popular, packets will have to travel further.

Pretty standard so far.

But Netflix is able to cheat and take it a step further.

You see, Netflix has control over the client. This gives them the ability to add intelligence directly in the client. The result? A system that can determine which place to grab the video content from, and which quality to use.

Here’s an overview of how it works:

• The API signals that the content this person is trying to watch is on, say, 10 different CDN servers.
• The client intelligence picks the CDN that it calculates to be best*
• The client intelligence continually probes to find the best way of receiving content

* It determines which CDN to pick by testing them all, determining latency, and continually running these tests throughout the video streaming process.

Ever notice the video quality drop quite a bit? Looks really pixelated and then, maybe a few minutes later, the quality is HD again?

That’s because it uses this technique to automatically switch between different CDNs and different bitrate levels depending on various factors.

That’s the (simplified) adaptive bitrate streaming concept. Continually testing to determine what sort of bandwidth is available in order to find the highest possible quality at the lowest possible latency.

If you found this interesting, you will probably enjoy Jeremy’s full interview.

There’s also a lot more where that came from. See more interviews.