Developing the New FOX Sports Digital Experience
With the 2021 Major League Baseball season starting this week, members of the FOX product development team reminisced about last season’s July Opening Day when we launched our new digital sports platform. The forthcoming nine-inning narrative barely scratches the surface of what was a very fun, challenging, and foundational year leading up to the 2020 season’s first pitch.
Top of the First Inning
The time had come to re-imagine the FOX Sports digital experience. The prior version of the FOX Sports website and mobile applications had been performing well in the market for several years but the technical and streaming landscape had changed, sports wagering was entering the market place and a new FOX had emerged. We needed to spring forward and lay the foundation for the next several years.
This was the rare opportunity to rebuild from scratch. For a product development team, this is what you dream about. Anything was possible and we had a hungry, talented tech team ready for the challenge.
Bottom of the First
As the business, product and design teams were focused on creating what the new experience should look like and how it should function, the rest of the product development team was focused on re-platforming decisions.
First, the mobile engineering team made an important decision to transition from cross-platform development to native development. Transitioning to native development (Swift for iOS, Kotlin for Android) would unquestionably yield many positives but it would also require the engineering team to learn new languages from scratch. The team dove in headfirst and spent several weeks ramping up on best practices, building out CI/CD pipelines, solution setup, key service and instrumentation integrations, etc. With a solid framework in place, the team would be ready to go.
Second, the technology team decided to adopt the company’s nascent, headless Content Management System (codenamed “Spark”). Sunsetting the legacy CMS would clear years of baggage and technical debt while also expanding the reach, feature set and adoption of the new Spark product which had recently launched to power the FOX Business digital experience. While it would eventually require training for the editorial staff, this decision paved the way for a single CMS system to power all the FOX digital products.
For the sports web development team, this meant a complete shift of technology. The team would transition from a blend of .NET/C#/Java/JavaScript developers to Vue and Nuxt. Like the mobile team, the web team was excited to learn new technologies.
For the CMS development team, this meant planning for an onslaught of new feature needs to satisfy a sports product.
Third, the technology team decided to unify the feature and task management system used across the various teams. Transitioning to JIRA introduced yet another learning curve for several of the teams. Select members were chosen to lead the charge, acquire knowledge and distribute to the team.
Finally, the teams all formally transitioned to a more consistent Agile/Scrum process with two-week sprints and the QA/test engineering teams were included as core members of the team from the very beginning.
While change is never easy and decisions are always questioned, hindsight has proven each was the right choice. Plus, the teams are now experts in each of the new technology choices.
Top of the Second
As designs started to progress, many unique new features were sprouting that would require outside partnerships or services to solve. Prior demo meetings, sales pitches, product research and existing relationships in other areas of the business started to bear fruit.
For example, the new digital product wanted to include third party content for both stories and video. Our partners at Are You Watching This, Rotowire and NewsWhip were able to help solve. We wanted to make social a big part of the new experience and we were able to expand our relationship with TagBoard to achieve. In addition to the tentpole data to power scoreboards, box scores, standings, etc., we really wanted to go deep on insights and our new friends at Sportradar proved to be great partners.
Bottom of the Second
A key aspect to the new FOX Sports digital experience would be video: live streams, full-length replays and short-form clips. Fortunately, we had an excellent in-house solution that was also busy innovating in advance of Super Bowl LIV. The video system already had just about everything we needed and was quickly becoming another best-of-breed product within FOX.
From the early design sessions, it became obvious digital-only live stream content and bonus streams would play an important differentiator in the new FOX Sports digital experience. As a result, the video team started planning how to deliver a single sporting event with multiple bonus streams with as little latency as possible. As if that wasn’t enough, there could be numerous sporting events with multiple bonus streams throughout the day and, sometimes, even at the same time. This introduced new challenges for the video team but, as you’d expect, they were excited to solve.
Top of the Third
As the web and mobile teams were busy learning new development languages, our backend team specializing on ingesting, augmenting, and exposing data for sports (aka the “SportsData” team) started to think about the optimal way to get the data to the clients.
Up until now, the website and native apps were very different beasts with few similarities. The client teams retrieved the necessary data from the SportsData API and wrote their own custom logic for functionality and display purposes. As you can imagine, there is very complex logic needed to render the myriad of views across a sports site where rules vary per sport. To rattle off just a few: live updating scoreboards, standings, player and team stats, box scores, odds, playoff brackets, play-by-play and polls.
The new digital experience would strive for consistency across platforms. This single decision provided the opportunity to greatly improve the flow of data, logic and presentation.
As a result, a new API middleware layer was conceived. Named after the bridge in Marvel’s Thor movie, Bifrost would be the bridge between the source data and the clients.
The new service would gather data from a variety of endpoints, massage, cache and deliver a set of templated responses the clients would interpret for optimal rendering. Business logic would be shifted to the server and clients would only need to worry about what endpoints to call and how to interpret the responses to render the UI.
A .NET Core microservice approach was used. Each sport would have its own scalable service. Prefetching of data could be introduced along with an intelligent caching mechanism based on feature type. Optimization opportunities continued to bubble up and the team enjoyed the analytic approach to coding a very efficient, powerful suite of microservices.
The decision to go with the Bifrost service required heavy coordination across the three client teams, the Bifrost team and the SportsData team. Many hours of meetings, Slack messages, emails and Quip documents led to a complete definition of response templates which would greatly simplify client development. As you can imagine, this was (and remains to this day) a fairly iterative solution where the teams communicate frequently to extend, tweak and improve the system. No client deployments necessary!
Bottom of the Third
Another initiative around this time was to consolidate all hosting to AWS. Several of our sports systems (including the legacy website) needed to migrate to AWS. Knocking this out was another important step in our rebuild flow as it allowed us to centralize and communize more of our systems and processes.
Top of the Fourth
At this point, designs for the new mobile app were far enough along for the teams to start focusing on key areas. Product managers were busy converting designs and annotations into user stories. Designers were busy cleaning up edge cases. The development team was busy taking in hundreds of design mocks to understand the big picture of what we were building. It was very exciting. As with any huge project, the key was whittling it down into smaller chunks and envisioning the large system, where all the pieces fit, and how they interacted with one another.
As stated, one of the goals was for both mobile apps and the website to be as consistent as possible. Using the layout manager component of the CMS was a great way to enforce this concept. The layout of every page could be dictated by a configurable collection of settings the clients could retrieve via API. Each “component” within a layout could include the external api endpoints to populate the component, tokenized string values for SEO, user-facing title text and more. Layout changes and endpoint adjustments could be made on the fly and clients would pick up the new settings in almost real-time. This was yet another powerful change that put more control on the servers to reduce the need for client releases.
Bottom of the Fourth
Introducing the concept of sports wagering was a huge focal point of the new suite of products. Up until very recently, most sports fans didn’t pay much attention to sports odds and wagering in the United States since it was only legal in Las Vegas. With laws changing, it was the time to begin showing our fans how much fun casual sports wagering could be.
FOX Sports’ relationship with FOX Bet was just starting. It included the free-to-play FOX Sports Super6 app where fans could compete for cash prizes by predicting outcomes during sporting events. It also included FOX Bet, a destination for sports fans to place wagers in the states with legalized sports betting.
The new FOX Sports app and website would include a combination of odds information ranging from the most popular markets such as spread, over/under and moneylines to individual team and player prop bets and futures. The challenge was to do it with a casual interface explaining what the information meant and for the game odds to update in real-time. We’d also need geo-targeted deeplinks so fans could start playing Super6 or place wagers in FOX Bet.
This effort required standing up a new line of services and tasks to ingest and expose betting data from FOX Bet. But the FOX Bet API didn’t exist yet, so we worked closely with their team to stand it up so we could consume it. This was challenging, but a great way for our team to gain a deep understanding of the betting landscape.
Top of the Fifth
As we hit November, 2019, we were full-steam ahead on building the new products. However, we had this minor sporting event coming up in February before we’d have the new product in the marketplace. That’s right, Super Bowl LIV on FOX was looming! It was at this time we had to carve off some of the team to focus on building out the Super Bowl experience on the legacy line of products. [Learn more about FOX’s Resilient, Low Latency Streaming Video Architecture for Super Bowl LIV.]
This was a significant challenge as we needed expert attention on both line of products. Fortunately, we had the needed talent depth to pull it off. And, as you can imagine, the teams were very excited to execute both successfully.
One of the mantras heading into Super Bowl development across the internal video platform and api team and client teams was to expect failure and build for resiliency. As a result, this approach was also adopted by the new FOX Sports digital products. Throughout the new mobile apps and website, we expect api calls to fail yet are always prepared to deliver our fans a great experience.
Bottom of the Fifth
With client teams developing what they could, the other teams were focused on ingesting data across various systems to support the new experience. This is the glue within the larger system.
Content, whether it’s video, articles or social posts, requires the correct metadata in order to be useful and map to the correct entities for display purposes. In the world of FOX Sports, our entities consisted of leagues, conferences, teams, athletes, events, shows and personalities.
An internally developed service nicknamed the “Relevancy Engine” helps us accomplish this. It’s capable of receiving any amount of data and suggests scored tags to associate with the content. When it’s a full article body, this can be pretty straightforward but when it’s limited to a title, one line sentence and a url, it can be much more difficult. It does its job with a combination of machine learning and custom logic to solve for the many nuances involved with sports. If this sounds easy, it just means you haven’t thought about it enough.
In addition, the CMS team was busy sucking in all it needed to allow for tagging throughout its system. This included the full hierarchy of sports data via our SportsData API. The CMS team also focused on consuming video clip metadata, third party news metadata, social post metadata, player news data, Associated Press content and much more. And with each different type of data pulled in, an accompanying addition was made to the CMS admin site and the API.
Top of the Sixth
While we have our own excellent internal mobile development team, we realized the workload and release deadline required more resources. We were able to augment our team with our friends at WillowTree. We grew the team substantially during this core stretch and it allowed us to increase velocity without sacrificing quality. The larger team allowed us to specialize in certain areas including the more complicated custom animations, header navigations, opening app sequence and more. The burst in resources also afforded us the opportunity to create an excellent automated testing pipeline and suite of regression tests.
Bottom of the Sixth
While teams were heavily focused on building out the core client features and support systems, there was yet another vital cog needed in the system. Fans needed to be able to favorite leagues, teams, players, shows and personalities and they needed to receive mobile alerts whenever something important happened.
Favoriting is, in general, a simple problem to solve. For us, however, it remained in a legacy system we lost confidence in during the 2019 Women’s World Cup. The service that housed favorites had encountered some issues during the highest concurrent traffic bursts. And there wasn’t a clear path to scale the system in its current state. So, we worked with our platform api team to solve the problem. They were able to quickly stand up a service to house favorites that would solve our sports problem but also pave the way to allow favoriting across the full line of FOX digital products. This was yet another great example of teams working together, thinking about the long-term goals of the entire organization, and solving with both the present and future in mind.
While we had a great system in place to generate dynamic data alerts for our prior app, we were expanding the reach for the new set of apps. We now wanted to send alerts at the athlete level and we wanted to expand our offerings. This was another big set of tasks put on the SportsData team and they ran with it.
Top of the Seventh
So far, we’ve discussed product development. But none of the product development team’s work matters if the FOX Sports editorial team isn’t creating excellent content. With the new digital experience, FOX Sports was bringing the written word back and it’d be front and center in the new app and site.
During this time, the editorial team was busy ramping up for the launch and its approach to stories. Each story would require an enticing custom image. Each story could include FOX Sports video clips, third party video clips, social posts, images, content tagging and much more.
The editorial and CMS teams worked closely together to optimize the publishing experience in advance of launch. This included a heavy amount of iterative development, testing and strategy evolutions.
Seventh Inning Stretch
Let’s now pause to remind everyone what was going on in the world in early and mid 2020. COVID was upon us and, for the first time ever, everyone needed to work from home. And the world of sports completely stopped.
Working exclusively from home was definitely a challenge but everyone adjusted to that pretty quickly thanks to the tools and services the teams were already using.
But no live sports…that presented a monumental set of challenges. While our SportsData system allowed us to simulate historical games to test many of the features we were building, there was no way to test many of the new features leveraging new data. We had no new data flowing through for the entire line of odds features or advanced insights we were introducing. We had to do our very best with mock API responses. Everyone understood the situation, contemplated various approaches and landed on a solution that eventually worked well enough to build confidence. But it was not easy!
Top of the Eighth
As we were nearing our launch date, we began putting the finishing pieces on the product. We were incorporating our first sponsor. We were load testing the various systems. We were documenting the most common user flows for our customer support and SRE teams so they could help as we rolled out the product.
However, there was one expected feature that hadn’t been included in our massive set of design mocks. And it proved to be a key feature to market the product. We needed simultaneous bonus stream playback to work in horizontal landscape mode in the app.
The team, fairly drained at this point, once again rose to the occasion and solved a very difficult technical problem in a short amount of time. The output was very impressive.
Bottom of the Eighth
While our development teams and QA teams were busy putting the finishing touches on comprehensive test plans, extending our automated testing coverage and bug fixing, our dev ops and site reliability engineering teams were focused on instrumentation.
Were logs going to the proper destination and in the correct format? Were all necessary events (automatic and custom) firing to the desired systems? Were the proper infrastructure alerts in place for each product and covering the desired set of metrics at the proper threshold levels? Were web redirects in place to handle the blend of new pages and sections while still maintaining the legacy site for some of the lower tier features?
The teams were also busy creating dashboards covering vital performance metrics, feature usage, crash details, etc. By the time the teams completed their work, we were confident we could monitor the entire system end-to-end and prepared to go deep on any issue that may arise.
In parallel, our data analysis team was heads-down creating reports and dashboards across our variety of reporting systems. The majority of the daily reports, monthly reports and one-off reports were in place before we launched so the teams could relay just about every aspect of user behavior.
Top of the Ninth
Another process that served us very well during Super Bowl LIV was the creation and adherence to a comprehensive run book that allowed us to methodically validate every step was completed in the correct order. We carried this concept over to our launch and it consisted of several weeks of activities that needed to be executed in a specific order since so many systems were involved. This approach contributed to what would be a very smooth release.
With the pending launch of the new set of digital products, we also overhauled our incident and escalation process. We adopted some new tools, Blameless and Pager Duty and merged with Slack to yield a clear escalation process when an incident occurred. To help test out this new process, we ran a serious of table top exercises to simulate real-world scenarios. Each exercise included the full set of teams and allowed team members to use the new system, discuss solution scenarios and play out incident scenarios across many facets of the business.
After several iterations, the team was confident in the process.
Bottom of the Ninth
The time had come. We officially soft launched on Saturday, July 18th. And everything worked.
Live streaming. Stories. Scores. Stats. Standings. Alerts. Bonus cameras. Video clips. League, Team & Player pages. Favoriting. Authentication. Profile creation. Sharing. Social Posts. Odds. Super6 integration. It all just worked.
That Monday morning, the onslaught of press releases were made and sports fans across the country were introduced to the new FOX Sports digital experience. MLB returned on July 23rd and the rest of the sports world returned soon after, all to be enjoyed in our new product.
One key trait we look for during our interview process is pride in ownership. Knowing your work will be enjoyed by millions is a huge motivator to create high quality, unique and reliable products. And on this day, the team was very, very proud!
Download the new FOX Sports App
or
visit the new FOX Sports Website.
