Road To Super Bowl 2023: Strategy for Migrating Legacy Services

The most important component any service needs to perform successfully, especially when undergoing changes, is a stable foundation. That’s why when we set out to increase our overall performance for our client-facing FOX services, we decided to rebuild all of them. In doing so, we could redesign a performant version of a service and compare the increase with its legacy service as a benchmark. In order to pull this off with zero service disruptions, we executed this in a dedicated months-long project that was designed to roll out in staged increments. This enabled us to redesign each service on a stable platform and optimize each service to meet our goals for a large scale event, such as the Super Bowl.

Figure 1 — Simplified Start State with one service as an example

Establish The Foundation

All of our client-facing requests transit through our AWS API Gateway and Load Balancer and are serviced by a collection of microservices, some of which have multiple routes. The client-facing services that we wanted to rebuild were read-only and shared a common database. So we focused our porting efforts on just rewriting the service without adding any new features, or changing the data model or the schema contract that we had with client-facing applications.

Our goal was to rebuild each service, route by route without taking on a massive and risky project to rebuild the entire microservices stack. The approach ultimately decided on was to introduce a pass through proxy service for each service we wanted to rebuild. This proxy service served as our scaffolding.

Figure 2 — Introduction of a Proxy Service as Scaffolding

Put Up The Scaffolding

We introduced the service specific proxy and performed a canary operation, routing a certain percentage of the traffic to the new proxy. When we were confident that the proxy was performing its role, we ended the canary and switched 100% of the traffic from the legacy service to the proxy, which in turn routed all its traffic to the legacy service.

Once the proxy scaffolding was in place, we embarked on rebuilding the legacy service, rebuilding each route of the legacy service and performing comparative tests on a route by route basis.

Figure 3 — Build New Service and Perform Testing

Test, Prove Success & Wave The Feature Flag High

Our testing criteria was that each new rebuilt route should be equivalent to the legacy route on the legacy service. Once our comparative tests showed equivalency and the service met our performance improvement goals, we introduced a feature flag in the proxy. This feature flag allowed us to canary a certain percentage of traffic to either the legacy service or the new service on a route-by-route basis.

Figure 4 — Feature Flag Enabling Canary to New Service

Removing The Scaffolding

Once we were satisfied with migration to the new service, we began our disassemblement of our scaffolding. First by removing our feature flag in the proxy and routing 100% of the traffic to the new service via the proxy.

Figure 5 — Full Traffic to New Service

Ready To Stand On Its Own

And as a final step, we would remove the legacy service from deployment and canary the traffic from our API Gateway to the new service, thus removing the proxy scaffolding.

Figure 6 — Newly Migrated Service

Lessons Learned

We performed these steps on each service that was part of our migration plan. This enabled us to take zero down time to route and to deploy all of our newly built services with a high degree of confidence that we would not disrupt any client-facing contracts or introduce any new issues.

The scaffolding ensured us a safety net, should anything have gone wrong, we would easily roll back traffic and stop our canary. But one of the biggest lessons learned when migrating an entire stack was to maintain the discipline of staying focused on the goals of the migration and to not introduce new features or changes to the data store or schema. This allowed the team to refactor the entire set of microservices on a solid foundation, focus improvements on a route by route basis, and enable us to prepare for large scale events that utilize our services at their full potential. Super Bowl 2023….it’s game on!

Be on the lookout for more from our Road To Super Bowl 2023 series and follow us on Medium so you’re the first to know when they’re available!

Interested in joining our FOX CPE team and helping deliver Super Bowl 2023 and more? Come join us — we’re actively hiring!