How does FanDuel technology win the Super Bowl?
Written by Matthew Clark — Software Engineering Director
The Super Bowl, which is the annual championship game of the National Football League (NFL) and the pinnacle of American football, is truly a global sporting phenomenon. With an estimated 200 million viewers in the United States alone tuning in to watch the latest championship game in February 2023, the sheer scale of this event demands that FanDuel’s technology stack is at its prime to ensure a seamless and memorable online sporting experience for our customers.
“We’re going to win Sunday. I guarantee it.” — Joe Namath (New York Jets, Super Bowl III winner)
Winning a Super Bowl requires meticulous planning and execution to maximize the chances of success — however the question remains — can we really guarantee a win beforehand?
One famous statement that boldly claimed a guaranteed win came from Joe Namath, the legendary quarterback of the New York Jets who led his team to victory in Super Bowl III. With his now-famous declaration, Namath expressed his unwavering confidence in his team’s ability to emerge victorious on that Sunday.
It’s important to remember that winning a Super Bowl involves much more than just confidence and bravado. It requires a well-coordinated effort from the entire team — from the coaching staff to the players on the field — all working together in perfect harmony.
So, while Namath’s guarantee of victory may have been a bold statement, it was ultimately backed up by the Jets’ hard work and preparation. And while we can never truly guarantee a win in any sporting event, we can certainly strive to give ourselves the best possible chance of success by putting in the effort and preparation required to come out on top.
Here at FanDuel, the Super Bowl is just as significant as any NFL team’s championship game. In fact, the groundwork for this highly anticipated event begins well in advance and involves the collaborative efforts of numerous teams of skilled individuals. Their shared goal is to ensure not only the best possible customer experience through our applications but also, most importantly, a seamless night for our customer support and on-call engineering staff.
We adhere to the mantra that practice makes perfect, and so we have built one of our primary Super Bowl preparation programs around this idea — Game Days. There are large-scale performance testing exercises that we conduct on our own production stack. Much like pitch side coaching staff, our Performance Engineering team at FanDuel is dedicated to supporting our engineering teams and product verticals. We provide the tooling and expertise required to support these exceptionally large, intensive dry-run events.
Let’s meet some of our MVPs from our team.
Locust as a Service — (Our distributed load testing tool using Kubernetes)
A few seasons back, we launched our internal SaaS platform: “Locust as a Service” — offering seamless ultra-large-scale testing using the Locust.io framework. This platform enables our engineers to create their own complex test scenarios and execute tests on their services and products, wherever they are deployed. This marked a significant departure from past methods, where we had depended on a single, static pool of hardware for testing. Most of this testing was driven by the Performance Engineering team, resulting in a bottleneck when queueing test runs that often-delayed crucial testing.
To offer this service to our internal teams we collaborated with our DevOps partners at FanDuel to establish a customised “Locust” Kubernetes cluster. Now, within minutes, multiple highly distributed testing deployments can be created with varying configurations, tests, resources, and more.
This approach proved to be a game changer. Within the first two months of launching our MVP version, we went from just one team or group of individuals (Performance Engineering) capable of running tests at one time to having over eight product verticals running tests with eighty-four individual deployments running concurrently. Another major benefit came from this setup in that we were able to support our Operational excellence program (GameDays). Like any NFL team we needed to know where we were at in terms of performance; by practising, we can help to identify our strengths, weaknesses well in advance of the big game. Without our Locust as a Service framework, we could not have been able provide the overall coverage required to ensure that we were properly match fit.
So, what are we practising for? Well, Super Bowl LVII in February 2023 saw a record number of bets placed across eighteen states over the 4-hour period with a max rate of several hundred of bets per second.
So, how can we adequately prepare for such a massive event and ensure that we can handle both the expected numbers and any unforeseen challenges? Just as we strive for success on the field, we need to go the extra mile in our preparations. In our last GameDay before the Super Bowl, our testing framework achieved the following results:
- We processed a maximum of 1.6k bets per second.
- Over eight million bets were placed.
- We successfully executed a total of four billion locust tasks. (Locust tasks refer to the operations performed by users.)
- The maximum number of locust tasks executed per second was 35.1k.
These achievements were made possible by the robust Kubernetes platform supporting us, which provided:
- A maximum deployment of 7k worker pods at any given time.
- A maximum deployment of 230 Locust clusters at any given time.
Locust Orchestration Tool
Complying with online sports gaming regulations in the United States presents a fascinating challenge for our infrastructure. To maintain compliance, we utilise a combination of in-state hybrid cloud solutions and traditional AWS cloud offerings, resulting in a unique production environment for each state in which FanDuel operates. As a result, running a gameday against production infrastructure significantly increases the number of individual test frameworks, tests, and the resources required to execute them. Managing the scale of running these tests across multiple states in a coordinated and phased approach is a challenge. For example: during the last Gameday before Super Bowl LVII, we had over 230 individual Locust deployments running. Managing the intensity of these tests and monitoring their progress at this scale becomes a significant challenge and is resource and time intensive.
Our Locust Orchestration tool was developed from our desire to empower our engineers with the freedom, time, and resources to experiment, learn, and problem-solve — providing innovative solutions to complex challenges. It embodies our commitment to listening to our internal customers and building upon existing functionality to enhance our overall performance testing solution. The orchestration tool is a way of managing a vast amount of Locust deployments in real-time from one single source of data. In-keeping with the FanDuel principle of “Simpler is Always Better”, our first iteration of this tool makes use of a humble Google Sheet to provide visibility, RBAC, API access and transparent logic.
To an outsider, our orchestration tool may appear as just a Google Sheet, but just serves as a dynamic data source used to provide the values required for the orchestration process. On a GameDay, a group of Locust deployments is controlled by an orchestration process that communicates with a master record of the required number of Locust users and the spawn rate of each Locust cluster at any given time. These parameters control the intensity and acceleration of the “user” load applied to services. This basic orchestration method makes use of a monitoring feature that was added to our framework, providing engineers with the ability to programmatically view and control various elements within a test run. For example, these hooks can be used to control the running test, send messages, interact with external data sources, and more, providing more flexibility and customisation in our Locust test suite.
The real winners from our orchestration tool are the stakeholders of those services that need to be tested within each state simultaneously. In some cases, we’ve been able to reduce the number of engineers required to run the tests by 75% which is a great efficiency gain and allows FanDuel engineers to concentrate on their own services closely.
The Performance Engineering team at FanDuel is dedicated to meeting the demands of growth and scale by constantly creating and evolving tools that make engineering easier for our product teams. Our primary focus is on ensuring overall product performance, stability, and reliability, which directly impacts the customer experience during both regular day-to-day operations and large-scale marquee sporting events.
While we cannot guarantee a victory akin to Joe Namath’s famous declaration, we understand that meticulous planning, execution, and continuous improvement are crucial for success. Through leveraging and enhancing tools such as “Locust as a Service” and our orchestration tool, we are committed to helping our engineering team “win every Super Bowl” by delivering optimal performance for our products.
