Resilient Deployment Strategies with Amazon ECS
Deployment strategies that are simple to execute, debug during failure, and mitigate worst-case scenarios
In 2023, AWS invited Perry Street Software to write a blog post documenting our experiences building devops deployment strategies on ECS. We were excited to share an approach that balances simplicity and resilience, and one that we feel is appropriate for small to mid-size development teams — aka >99% of the actual teams that are using AWS.
Below is the introduction to our blog post. If you’re interested, please read more at the official AWS blog!
You just finished deploying that important change you spent weeks preparing, when you see this email subject in your inbox: Alarm: HTTPCode_Target_5XX_Count.
Ugh.
The code you have just deployed is causing a critical error in your application server. As dev ops professionals, we are told to just cut back over to the original code, and indeed this very sensible advice. But buried in that advice are two discrete, time-consuming steps: first, removing malfunctioning code from an Application Load Balancer, and second, adding previously functioning code back into the Application Load Balancer. Both steps take precious seconds or minutes, during which your customers are unable to use your application, and, in the worst scenarios, data is being lost or corrupted.
If you think about it, code deployments are like dating. Imagine a cluster of application servers: deploying new code to 1% is the equivalent of chatting online, 2% is coffee, 5% is drinks, 10% dinner, 50% is meeting the parents, and 100% is marriage. Deployment strategies like blue/green are effectively advocating for complete commitments, followed by sudden divorces, with your application server software versions. It’s a relationship strategy that can work for some people, but not without a lot of heartache.
Perry Street Software (PSS) is home to SCRUFF and Jack’d, two of the world’s largest gay, bi, trans, and queer social dating apps on iOS and Android. Perry Street’s brands reach over 30 million members worldwide so members can connect, meet, hook up and express themselves on a platform that prioritizes privacy and security. PSS is involved in internet policy at a national level, and has engaged with Senators, Congresspeople and policymakers on Section 230 and platform regulation to help defend LGBTQ+ digital spaces.
For its entire history, PSS has had a lean ops team that supports an active user base 24 hours a day, 365 days a year. Because of the global nature of the dating app business, downtime, even in the early hours of the morning, is not an option. Finding ops deployment strategies that are both simple to execute, simple to debug during failure, and mitigate worst-case scenarios are critically important.
Eric Silverberg is CEO of Perry Street Software, publisher of the LGBTQ+ dating apps SCRUFF and Jack’d, with more than 30M members worldwide.
