CentOS 7 Exits, Debian Rocks: How IPSY Survived a Linux Soap Opera
Introduction
At IPSY, more than a year ago, we faced a drama that would make any infrastructure team’s heart race: the end-of-life announcement for CentOS 7 on June 30, 2024. This plot twist launched us into a mission-critical operation to transition our systems to a new operating system without disrupting our services — a task easier said than done. Welcome to our tale of twists and turns in the world of Linux.
In this article, we’ll take you behind the scenes of our decision-making process, technical implementation, and the surprising perks we discovered along the way. Whether you’re navigating a similar transition, curious about tech infrastructure updates, or just love a good tech story, our experience is packed with insights for you.
We’ll cover:
- The factors that influenced our choice of a new operating system
- The nitty-gritty technical details of our implementation, including some challenges we faced
- The gains in efficiency we achieved
- Lessons learned that might help other organizations in similar situations
Join us as we unravel the tale of our migration — a story of careful planning, technical hurdles, and ultimately, a more robust and efficient infrastructure.
Decision Making
When Red Hat announced the shift of CentOS to CentOS Stream, positioning it as an upstream development platform for RHEL, we knew it was time to reevaluate our options. Our initial plan was to transition to Rocky Linux — another RHEL-based system like CentOS 7 — with Gregory Kurtzer as the CEO (one of the original founders of CentOS) and an important sponsorship from very well known companies. However, Red Hat’s changes complicated access to updates and patches, making Rocky Linux less appealing.
We then turned our attention to Debian, a stalwart in the Linux community known for its stability and extensive package ecosystem. Debian’s commitment to free and open-source software and its robust update mechanism made it an ideal choice. Despite significant efforts already put into Rocky Linux, Debian emerged as the hero we need in this tech drama as a more reliable and future-proof option for our needs.
Technical Implementation
Our migration process involved several technical enhancements and changes to our existing workflows:
Automation with Packer and Ansible: The Dynamic Duo
Our CentOS 7 custom AMIs were previously built using Packer templates with Puppet for configuration management. For the Debian migration, we switched to Packer templates combined with Ansible. This shift allowed us to automate the configuration and management of our AMIs more efficiently using Ansible’s simple, human-readable YAML syntax. Moreover, this change enabled us to adopt a new approach: instead of continuously reconfiguring our servers with Puppet, we now build golden AMIs. We will explain this approach further later in the article. Additionally, we moved from Packer’s JSON syntax to HCL, aligning with our familiarity with HCL from our Terraform usage.
General Purpose AMIs: One Size Fits None
We moved away from using a single general-purpose AMI to creating custom AMIs for specific runtimes (aka golden AMIs), such as Node.js, Python, and Java, each with their respective versions. This change enabled us to optimize performance and resource utilization for different applications, ensuring each runtime environment was finely tuned for its specific use case. This approach significantly accelerated instance launch times.
Cloud-Native Tools and Elimination of Package-Syncer
Our previous setup involved a cumbersome package-syncer process that mirrored packages from official repositories to our S3 buckets. By embedding all necessary packages directly into our AMIs, we eliminated the need for package-syncer. This simplification ensured that our instances were fully configured and ready to use immediately upon launch, streamlining our deployment process.
Embracing Envoy: A New Era in Proxying
We made a strategic decision to replace Apache with Envoy as our reverse proxy. Envoy, an open-source edge and service proxy, offered enhanced capabilities such as advanced load balancing, observability features, and seamless service mesh integration. This change positioned us to better handle our growing microservices architecture and improve overall system performance.
Terraform Integration: Fast & Furious Migration
At IPSY, we’ve developed and maintained custom in-house Terraform modules to manage our infrastructure. Because the infrastructure team loves their terraform modules more than Dom Toretto loves his family, it was decided to smooth the transition to the new Debian-based AMIs, ensuring that our terraform modules could support the new AMIs without introducing breaking changes.
By just adding few variables, for the runtime (e.g., runtime = “nodejs”) and another for the runtime_version (e.g., runtime_version = “18”), we made it easy for our launch templates to pick the correct AMI based on the application requirements. This approach allowed for a gradual transition, enabling teams to adopt the new system at their own pace while maintaining backward compatibility.
Challenges and Lessons Learned: The Plot Twists
No good story is complete without a few plot twists, and our migration tale is no exception. Here are some key takeaways from our journey:
Stay Informed and Agile: The linux Soap Plot Twist
Knowing about the EOL (End Of Live) of your OS is just the beginning. It’s crucial to stay attuned to the broader community and industry trends. The Red Hat announcement about repositioning CentOS Stream as an upstream development platform for RHEL was a game-changer, pushing us towards Debian. This experience underscored the importance of remaining flexible and ready to adjust our strategy based on evolving circumstances in the open-source ecosystem.
Embrace Opportunities for Improvement: The Makeover Episode
Integrating Ansible with our existing Packer templates required a complete overhaul rather than incremental changes — think “Extreme Home Makeover: Infrastructure Edition”. While this presented a significant challenge initially, it also offered an opportunity to build a more robust and flexible automation framework from the ground up. This fresh start allowed us to incorporate best practices and modern tooling, resulting in a more efficient and maintainable configuration management process.
Maintain Fallback Options: The Back Up Plan
During our transition, we made the strategic decision to continue AMI development using Rocky Linux alongside our Debian efforts. This approach served two purposes: it provided a fallback option in case of unforeseen issues with Debian, and it allowed us to leverage the work we had already invested in Rocky Linux. This dual-track approach gave us valuable insights into both systems and ultimately reinforced our decision to adopt Debian fully.
Performance Improvements: The Fast and the Efficient
Our migration process was designed with a focus on speed and efficiency. By moving away from Puppet and adopting a more streamlined approach with custom runtime-specific AMIs led to significant improvements in deployment times:
Auto Scaling and CodeDeploy Enhancements
Our Auto Scaling Groups and CodeDeploy processes were optimized to be up to 11 times faster compared to the previous CentOS 7 setup. This dramatic improvement not only reduced downtime but also enhanced our overall operational efficiency. The shift to runtime-specific AMIs, pre-loaded with necessary dependencies, allowed for faster instance launches and more rapid application deployments.
Conclusion
The migration from CentOS 7 to Debian was a strategic decision that brought numerous benefits to IPSY. By leveraging Packer and Ansible, creating custom AMIs for specific runtimes, and simplifying our package management process, we achieved a more efficient, secure, and scalable infrastructure. This transition ensured compliance, enhanced security, and positioned us for future growth.
We hope that by sharing our journey — complete with its unexpected twists, technical triumphs, and even few facepalm moments — we’ve provided some valuable insights for other organizations facing similar transitions. The path from CentOS 7 to Debian was not always smooth, but the destination proved well worth the journey. As we continue to evolve our cloud infrastructure at IPSY, we’re excited about the possibilities that lie ahead and the strong foundation we’ve built for future innovations.
