Getting to Ultra Low Latency with Video

The next time you tune into a YouTube Livestream, Facebook Live Event, or Twitch Broadcast, remember one thing — live doesn’t mean live.

What does that mean? It breaks down to the fact that while the event you are watching is in fact currently live, the time it takes to encode the video into chunks, pass those chunks to the internet, and have the internet pass them to you is what we call latency.

If you take a step back, brands have regularly sacrificed latency in order to gain in other areas. In the flash era, we were helping customers deliver their broadcasted content to users over the internet using Flash Media Server. This made use of RTMP/RTSP protocols, which could produce latencies of sub 2-seconds if deployed correctly. Deployment issues, Apple ultimately killing flash, and high CDN costs introduced us, and the rest of the Media industry, to HTTP Streaming technologies.

HTTP Media Streaming has been great, it reduces bandwidth costs, allows us to switch bitrates mid-stream to reduce buffering, allows for better ad integration, and better encryption support. Those gains come at the cost of increased latency. See before, using Flash Media Server, we would just open a constant connection to the users device and push data through. Now we need to split our media into “chunks”, encode those chunks to a specific format and packaging, push those to the CDN, and have the CDN deliver them to you. If we add in the step of getting the broadcast from the camera to the encoder, this is what we refer to in the industry as “glass-to-glass” latency.

With application requirements getting more interactive and inclusive of an audience, the entire industry has been making moves to create workflows that decrease the latency we took on with HTTP Streaming. For example, to provide a live video experience where the latency is low enough that the viewers can feel the live interaction with the host, we considered the following:

• HLS and DASH would be the most ideal, but common latency for glass-to-glass trends around 7–12 seconds.
• RTMP has the latency required, we covered that, but we lose so much and increase our costs as most CDNs have began deprecating RTMP.

Research in the community

Over the past year, there have been large leaps made to support ultra low latency over HTTP Streaming; this includes showcased demos by a few open source vendors. Some of the largest media giants and infrastructure-as-service providers have become aware of the need as well.

• CMAF is a community driven standardization for a common packaging format regardless of the protocol. This means we can use either HLS, DASH, or both and work with our CDN partners to provide CMAF support.
• A great white paper published out of ParisTech outlines a proof of concept using DASH delivery with chunked-transfer, and some updates to the DASH manifest specification allows video players to be aware of segments before they were complete and request them as they’re still being published from encoder to origin. This POC resulted in a latency sample of ~1.5s on various network conditions.

Our Proof of Concept

After doing some of our own research, and going through the ParisTech white paper, we felt the best path forward was to make use of CMAF packaging, chunked-transfer, and test both DASH and HLS protocols. This required resolving a few dependencies first:

• CDN providers needed to support both chunked-transfer and CMAF packaging.
• We needed a hardware encoder that supported CMAF packaging.
• Was there any iOS and Android players that supported chunked-transfer natively?

We worked with both of our CDN partners to educate their teams and test their CMAF implementations on their platforms. We found and partnered with hardware encoding company Keepixo, which is one of the few hardware encoders we could find on the market that supports CMAF packaging. We worked with them, testing their ultra low latency spec and helped write proxy code to authenticate with our CDN origins.

For video players, we tested many. Some who claim to already support ultra low latency (basically enabling chunked-transfer and some buffer configuration) and others who were willing to partner with us to implement it. To date, our most stable implementation has been a custom build of both AVPlayer and ExoPlayer.

Proof of Concept results

For our POC, we’re testing with a bitrate of 1 mbps which gives us pretty good quality when working with mobile resolutions. We understood the results published in the white paper were subjective to local network conditions. Prior to doing network stability testing and some buffer configuration our results for glass-to-glass resulted in average latencies of 3 seconds. We found no substantial edge latency difference between either of our CDN partners after fully implementing CMAF packaging support.

Moving Forward

3 seconds is crazy! To be able to offer this to our clients is an awesome achievement that we’re going to continue to work on. We tackled mobile first, whereas others have recently rolled out this latency support on Desktop Browsers only. We know having direct access to how we fetch chunks in the browser will be far easier than mobile and with our latest round of network stability testing we have drastically increased our stability without sacrificing latency.

REDspace helps global brands deliver content to every device. We can now bring Ultra Low Latency Video to your workflow. Let us help — redspace.com.