Road to Super Bowl 2023: Improvements to FOX’s resilient, low latency streaming video architecture for Super Bowl LVII
It’s been almost a year since we published our streaming video architecture for Super Bowl LIV in 2020. As the ball gets handed to us for the next Super Bowl, we thought it might be ideal timing to share some of the improvements being planned for what we are calling our “Season of Champions”, including the following marquee events:
- 2022 MLB season & World Series
- 2022 FIFA Men’s World Cup
- 2022 NFL season & NFC Championship Game
- 2022 Midterm Elections
- 2023 Super Bowl LVII
- 2023 FIFA Women’s World Cup
Busy times ahead for sure!
In preparation for these major events, our team is starting early and is already lining up work for our engineering teams to build out the video workflows and players to support these large scale and high profile events.
This article will cover the following topics:
- Transmission improvements
- Encoder settings optimizations
- Infrastructure automation
- Observability improvements
- Multi CDN decisioning improvements
- Player improvements
1. Transmission Improvements
For Super Bowl LIV in 2020, we had built a highly redundant transmission pipeline. Each incoming contribution signal was an RTP-FEC feed with video at 720p60 20 Mbps, AVC. This time around, we’re looking to utilize the state of the art infrastructure from our broadcast facility to upgrade the feed to a 1080p60 HDR source. We’re exploring the following open questions for the contribution feed:
- Codec format: Do we use AVC or upgrade to using HEVC for the contribution feed?
- Bitrate: Based on the codec choice made, what’s the optimal bitrate to choose for the source signal?
- Transmission Protocol: Do we stick to RTP-FEC, or do we go with an option like Zixi?
- Optimized Transmission Path: How do we minimize the number of transmit hops that the contribution feed makes before they hit our geo-redundant cloud encoders in multiple cloud regions?
- Resiliency: How do we build additional diverse backup feeds to our cloud encoders?
2. Encoder Settings Optimizations
To deliver an optimal end user experience, we need to optimize the adaptive bitrate ladder for a wide variety of devices, while keeping CDN bandwidth utilization in mind. Other considerations include:
- Number of steps in the bitrate ladder
- Peak bandwidth for each rung of the bitrate ladder
- GOP size
- Sync lock settings for seamless failover within/across regions
- 5.1 Audio
- Support for the widest variety of devices
3. Infrastructure Automation
We have built our video workflow using infrastructure as code and are currently expanding our automation suite to spin up and tear down entire video workflows as described in the diagram below:
4. Observability improvements
A key initiative is to better understand our end user Quality of Experience (QoE). We use several industry standard player QoE tools to understand our user engagement and QoE. We combine player QoE tools with CDN logs and synthetic monitoring tools to successfully operate large scale live events.
Today when alerts occur, it is cumbersome to debug and understand which part of the ecosystem is the root cause of the issue. For instance, if you are an end user watching an NFL game on Sunday afternoon and experience rebuffing, is it due to slow origin responses? Or caching inefficiencies at the CDN? Or poor buffer management in players?
We typically look at player QoE tools to diagnose the issue and then look at CDN logs to further validate, if required. Hence debugging a single session involves logging into multiple tools and correlating the data manually. This is where CMCD comes into the mix.
The Common Media Client Data (CMCD) is a specification developed by the CTA-Wave. CMCD provides a ‘consolidated’ view of metrics as observed from both the clients (players) and CDNs. Players report a standardized set of QoE tracking data for every CDN request. This tracking data can subsequently be captured and viewed in CDN logs. Players can send this tracking data either as HTTP headers or query string params on CDN object requests.
The CMCD specification can be found here.
Planned work involves:
- Updates to our player frameworks to send CMCD data to the CDN
- Have CDNs log CMCD data to our log aggregation tool along with other standard metrics (time to first byte, cache hit ratios, time to last byte, average round trip times, etc)
- Build dashboards to monitor and debug on a per session level
Key metrics for tracking:
This will give us the ability to visualize individual sessions, triage problems and perform mitigation steps.
5. Multi CDN Decisioning Improvements
For Super Bowl LIV, we built a backend service for CDN decisioning with the decision made at session start. Additional data vectors under consideration:
Synthetic Monitoring Data
- Synthetic monitoring involves periodically sampling the master manifest, the child manifest and segments across multiple CDNs. The sampling is done from a few geographically distributed locations. This gives us additional insights into throughput and any potential errors on a per CDN basis. Funneling back this data to the CDN decisioning engine will give us an additional dimension for recommending a CDN at session start.
Additional Player QoE data
- Utilizing player QoE data from additional aggregator tools will give us additional data for decisioning.
6. Player Improvements
Another key initiative is striving for consistency of QoE implementation across our various players, enabling focus on iterative improvement to overall quality.Our player teams are currently working on:
Consistent and More Verbose Error Reporting:
We are also working on improving our player QoE implementation to add more verbose and structured error reporting. Instead of reporting generic “Playback failed” errors, we will be adding descriptive error messages explaining the reason why a video start failure or a video playback failure occurred
Consistent Retry Logic Across Players:
Retry logic in players can be tricky. Important considerations are:
- Specific error conditions for retry logic.
- Interval and number of retry attempts.
We’re working on smart algorithms for doing reties without elevating rebuffering or video playback failures.
Overall, lots of interesting challenges ahead for the team. And by the way, if you’re interested in joining us to help build and solve some of the challenges described above… we’re hiring 😀
Be on the lookout for more from our Road To Super Bowl 2023 series and follow us on Medium so you’re the first to know when they’re available!
About FOX Tech
Make Your Mark Here. At FOX, we pride ourselves in shaking things up and making things happen. We’re a community of builders, operators and innovators and each and every day we experiment, collaborate, and co-create to develop the next world of news, sports & entertainment streaming technology.
While being one of the most well-known brands in the world, we provide our employees with the culture of a start-up — fast paced, non-hierarchical, full of smart ideas & innovation and most importantly, the knowledge that each member of the team is making a difference in defining what’s next for FOX Tech. Simply put, we love to do great work, with great people.
Learn more and join our team: https://tech.fox.com
