Part I — Behind the Scenes: The 2023 FOX Sports Super Bowl LVII Stream
Working as a Program Manager for FOX Sports means you’re always in a cycle of planning and delivering the next set of features for our apps every few weeks. However, every two to three years something truly special comes along, something of unprecedented scale that compares to no other work we do at FOX Sports or well, at most jobs: hosting the Super Bowl live stream. Virtually every year, the Super Bowl is one of the most streamed live events of all time and the 2023 Super Bowl was no different. Super Bowl LVII in February had a peak audience of over 7 million streams, making it the most-streamed Super Bowl ever and most streamed event in FOX Sports history.
The Super Bowl LVII stream was a resounding success for the FOX Tech team. Beyond breaking all kinds of streaming and platform records, we had zero critical incidents and an 81% decrease in our total incident count compared to the 2020 Super Bowl. Our live stream latency was vastly improved, even beating cable in certain instances (source). Some team members dared to use the word “boring” to describe our Super Bowl 2023 war room, especially compared to the one in 2020. However, a “boring” Super Bowl Sunday war room didn’t happen by accident. It took an army of smart people all working together towards one common goal: to deliver the Super Bowl LVII live stream flawlessly to millions of people.
This article won’t go into the technical details of the architecture, scaling, platform, and app updates needed to support 3X the traffic of our biggest 2022 World Cup game and 2X our previous platform peak from the 2020 Super Bowl. That’s for another article that’s soon to follow. It will instead focus on the months of preparation spent to ensure that our teams and tech stack were prepared for the most streamed Super Bowl in history — because it turns out, the NFL won’t move the Super Bowl date, no matter how nicely you ask. That’s where the Program Management team at FOX Tech came in to bring it all together.
Why Streaming the Super Bowl is Hard
The biggest and most obvious challenge with hosting the Super Bowl stream was handling the scale. There’s no other event we stream at FOX Sports that comes remotely close to the amount of streaming traffic for the Super Bowl. For a typical NFL Sunday, our infrastructure team scales up to handle the projected traffic and our site reliability operations team runs the event, addressing any issues that may come up.
As with all things designed to push the envelope, there are some next level scale challenges we plan to address before the next Super Bowl in 2025. These learnings come from understanding our own limitations from this past Super Bowl. Scaling up our infrastructure to handle the projected traffic was not as simple as it seemed – even pushing our biggest dynamic cloud services to the max would not have been able to handle the projected traffic. To address this, we had to create across our tech stack what we called “Super Bowl Mode”– heavily caching our APIs and judiciously enabling features that hit our infrastructure hard. To create this mode, we needed to release coordinated new builds across our full tech stack and update all 21 FOX and FOX Sports apps where viewers could stream the game. This required involvement from the vast majority of our teams at FOX Tech–all 250+ of them scattered across the globe. In other words, an absolute program management behemoth and nightmare.
How We Got Ready: Super Bowl Simulations
To tackle the problem of building “Super Bowl Mode” across our full tech stack in a short timeframe, we hosted a series of Super Bowl Sunday simulations that compressed the 12+ hours of Super Bowl Sunday programming into 2–3 hours. They were scheduled well in advance, which created a release-train cadence for all teams to come together to test their latest “Super Bowl Mode” code.
What we didn’t want to do is apply a “waterfall” project management style to our Super Bowl Sunday simulations by waiting for everyone’s development to be completed first. Had we waited until all of our builds across teams were ready in early January, we would have been in quite a predicament. We needed to find the biggest issues and “gotchas” as soon as possible while we still had time to fix them. Instead, we followed an Agile approach and scheduled our Super Bowl simulations starting in early December 2022, well before our teams were “dev complete”.
Using incomplete builds meant some features wouldn’t be available and other areas of apps would be flat out broken. Each of our simulations had 60+ team members on Zoom for multiple hours at a time. Considering the hourly price tag of meetings, these simulations weren’t cheap, so we needed to run them as efficiently as possible.
Much like conducting an orchestra, putting on successful simulations required making sure all the teams had the info they needed to play together — or we would quickly devolve into chaos. Before each Super Bowl simulation, we summarized the goals, which new features were ready, what we were doing differently this round, and what we knew would be broken. The most important piece of each test was the run book– a minute by minute plan of the most critical steps that needed to be taken across teams, including when we would enable “Super Bowl Mode” on all of our apps, the timing of when the pre-game or game would start, and specific features we wanted to cover like push notifications or game listing extensions. The major steps were announced over the Zoom bridge so everyone knew what was coming next and when they needed to take action.
Using Chaos Testing To Expect the Unexpected
Having a plan is key, but nothing ever goes according to plan, especially on Super Bowl Sunday. We needed to make sure our teams were ready for the unexpected. That’s where chaos testing came into play. During most of our simulations, one of our tech leads broke something catastrophic on purpose to see how the team would react while under pressure — e.g. a simulated AWS region failure or a CDN provider going down. Then we’d sit back, enjoy our popcorn, and watch the chaos unfold.
We measured how long it took for someone to report an issue and if reports were taken seriously. We observed whether or not a tech lead took ownership to mitigate the issue as quickly as possible and how long it took from when the incident occurred to when it was resolved. We made notes on how communication on the Zoom bridge was handled and if the broader team was clear on its status. Most importantly, we adapted.
Although it felt somewhat cruel (yet selfishly entertaining) to have our teams always on edge, worried about what was going to break next, our chaos testing paid off with huge dividends. After each simulation we held a retrospective to discuss how the incidents were handled and what to do differently next time, which resulted in dramatic improvements. Together we:
- Improved how we used Zoom breakout rooms and tracked incidents
- Established dedicated team roles to handle multiple critical outages at once
- Improved our test automation and health dashboards to better identify issues before they surfaced on our apps
- Wrote scripts to replace manual failover steps to enable failover states automatically.
All of these learnings helped us be ready for whatever could come our way on Super Bowl Sunday.
In the end, we ran 17– you read that right, 17– end-to-end Super Bowl Sunday simulations across our lower environments and in Production with real live sporting events. Production simulations raised the stakes; it’s one thing to break something for our internal team at FOX, it’s a whole different ball game (literally) if a diehard fan can’t watch the game. Still, we needed our team members who typically didn’t work live events to gain experience dealing with real viewers and real consequences if something went wrong. While there were some bumps along the way (apologies to Bob from Michigan who couldn’t check his college hockey scores during a college basketball test we ran!), we didn’t want our first time testing everything in Production to be Super Bowl Sunday. No matter how close to Production your lower environments are, they will never exactly mimic your Production environment with real users across the country. Through our series of Super Bowl simulations, our team became a well-oiled machine, ready for anything that could come our way on game day.
Super Bowl Sunday, The Results, & Getting Ready Again for 2025
When Super Bowl Sunday finally came, there was a palpable sense of nervous excitement across the team. We’d been working towards the big day for months. After working purely over Zoom for several months, a good portion of the team finally met in person for the first time at satellite offices in Tempe, Los Angeles, Cincinnati, New York, and Bangalore for closer collaboration. We knew we had prepared as much as we possibly could, but we only had one shot to get it right and the stakes couldn’t be higher. It wasn’t long before our first curveball came when we discovered a few APIs that hadn’t been cached and could topple over our infrastructure when traffic ramped up. But the team handled it like a champ. We identified a solution and pushed an emergency code deployment well before our traffic started to spike. From there, it was smooth sailing.
We watched traffic ramp up through the pre-game shows. An hour before kickoff, we were already past our peak stream viewership from the 2020 Super Bowl! We watched the hockey-stick growth of stress leading up to kickoff while keeping a close eye on our health dashboards, social media, and the customer care team — all was quiet! Throughout the game, there were only a handful of low-impact incidents despite hitting a record-setting 7+ million peak concurrent streams. By the time the Super Bowl post-game show was over, we were ready for a very well deserved champagne toast. All of the planning and preparation had paid off!
Hosting the Super Bowl stream is kind of like landing on the moon. It exposes the weaknesses in your platform that you need to overcome and creates major leaps in technology in a short period of time. As we get ready for the Fall 2023 NFL season, we have never been in a better spot to provide a delightful experience to our fans based on what we learned and the platform improvements we made for the Super Bowl LVII live stream. Time to do it all again in less than two years!
ABOUT FOX TECH
Make Your Mark Here.
At FOX, we pride ourselves in shaking things up and making things happen. We’re a community of builders, operators and innovators and each and every day we experiment, collaborate, and co-create to develop the next world of news, sports & entertainment streaming technology.
While being one of the most well-known brands in the world, we provide our employees with the culture of a start-up — fast paced, non-hierarchical, full of smart ideas & innovation and most importantly, the knowledge that each member of the team is making a difference in defining what’s next for FOX Tech. Simply put, we love to do great work, with great people.
Learn more and join our team: https://www.foxcareers.com/
