Quantifying our benefits of using a Blue-Green Delivery strategy

By: Pramod K, Associate Director, SaaS Customer Service; Matt Hefford, Head of Cloud Services; Jijoe Vurghese, Head of Platform Engineering.

Introduction

The Apple iPhone, introduced in 2007, caused a seismic shift in our expectations of software. Have you noticed that we’ve come to expect frequent updates to our apps? Furthermore, this expectation leads us to assume (rightly or not) that an infrequently updated app is abandonware. This expectation extends to SaaS products and platforms.

Deployment frequency is one of the four measures of organization performance identified by Google’s longest-running research by the (DORA) team. However, teams must balance deployment speed with system availability.

Continuous Delivery is a crucial tenet of lean, agile value delivery while maximizing system uptime. As Martin Fowler put it succinctly, “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.”

Banking regulations require explicit customer acceptance and approvals for changes to products. Zafin continuously delivers our SaaS product updates to customer validation environments pending customer acceptance. Subsequently, “continuous” delivery techniques complete the last-mile delivery into production environments.

We have several tools in the tool chest to achieve Continuous Delivery, including blue-green deployments, canary releases and feature toggles. In the fast-paced world of software development, deploying updates and new features to production environments requires careful planning to avoid downtime and potential service disruptions.

Blue-Green Deployment has emerged as a powerful strategy that enables organizations to release software changes while maintaining continuous service. As we continue our GitOps journey, we will dive into the benefits of Blue-Green deployment and how it is effectively implemented in Zafin to ensure smoother and more reliable software releases.

What is Blue-Green Deployment?

Blue-Green Deployment is a software deployment strategy used in the field of DevOps and continuous delivery to minimize downtime and risk during the release process of a software application or service. The primary goal of this approach is to ensure that the latest version of the application is deployed and tested thoroughly before it is made available.

In a Blue-Green Deployment, two identical production environments are maintained, each with its own set of resources, and infrastructure.

Case Study on what and how we did it in Zafin

Zafin’s cloud deployment architecture on Azure was lacking Blue-Green deployment capability. This resulted in

• Service non-availability/downtimes during platform updates and application deployments.
• Uncertainty around downtime required for maintenance.
• Pressure on deployment managers to execute the maintenance activity in a stringent, contractually agreed maintenance window.

The significance of 24/7 service availability increased when Zafin’s clients moved away from batch mode processing to more of real time data processing via APIs and Kafka events.

To address the challenges and changing needs, we decided to migrate to a modern, cloud-native infrastructure that supports Blue-Green Deployment. The goals are, achieve zero downtime during deployments and updates, improve application performance, and enhance overall experience.

Figure 1 illustrates our deployment process.

Figure 1: Our Deployment Process

1. The existing production application runs in the Blue environment, serving current traffic.
2. In the event of a new application version release/upgrade, an identical Green environment is provisioned in the existing Kubernetes cluster.
3. Traffic routing is managed by Azure Traffic Manager (Global DNS Load Balancer), allowing seamless switching between Blue and Green environments without impacting availability.
4. The basic sanity test of the application will be done in the Green environment.
5. Once the Green environment is deemed stable, the traffic switch in the Azure Kubernetes platform will be initiated, directing incoming traffic to the Green environment.
6. The Green environment becomes the new production (Blue) environment.

Pre-Requisites

1. Ensure that the following roles have been configured and are active before the deployment starts:

a. Azure Kubernetes Service RBAC Admin

b. Azure Kubernetes Service Contributor Role

2. Enable project access to the Azure DevOps pipeline.

Pre-Deployment Steps:

1. Perform a sanity check on the Application UI and API availability.
2. Take appropriate data/config map backups.
3. Review the below steps with the Cloud Engineering team

a. Infra Sizing

b. Confirm that the environment is running the latest Kubernetes version.

c. The migration path is correctly configured.

Deployment Steps

1. Current state

2. Terraform changes to the below configuration:

• This will ensure the Green pods are up and running while the traffic is still being served by the Blue.
• Compare the config maps and docker images with the Blue to ensure that all the properties are in sync. Any missing properties are to be added to the green pods.

3. Run a quick check to ensure the Application UI/APIs are available for the Green pods by using the Traffic manager.

4. Run the implementation specific DB scripts, deploy implementation changes to the Green pods and ensure the execution is successful.

5. Perform all the necessary sanity checks on the Green deployment (Application UI access, API connectivity, etc.).

6. Run the ‘Phase 3’ pipeline, that will switch the traffic from Blue to Green pods, while the Blue will still be up and running. A UI/API check is mandatory here to ensure that post the switch, all services are up and running.

7. Finally, run the ‘phase 4’ pipeline that will scale down all the Blue pods. Post this step, there will only be Green pods. The following is the pipeline configuration for Phase 4:

Perform a UI/API sanity check and check for any downtime during the switch over process.

The Benefits

• Fully Automated Deployment: With the cloud-native infrastructure in place, we could set up a robust Continuous Integration and Continuous Deployment (CI/CD) pipeline using Azure DevOps. The CI/CD pipeline automated the entire deployment process, which enabled the deployment team to push updates to these environments effortlessly. Kubernetes orchestrated the deployment process, ensuring consistency and scalability across the application’s life cycle.
• Improved Application performance: Auto scaling was implemented in the Kubernetes cluster to address the dynamic user requests/load. With auto scaling feature, the infrastructure could dynamically adjust its resources based on demand. This eliminated the need for manual intervention during traffic fluctuations, improving application performance.
• 24x7 Service availability: Azure’s managed services efficiently handled infrastructure patch upgrades without any downtime for the customer’s application. By leveraging managed services, such as Azure Kubernetes Service (AKS), Zafin could focus on building and deploying their application while Azure took care of underlying infrastructure updates and maintenance.

Lessons Learned

1. Communication and Collaboration: Successful migration to Blue-Green technology necessitates close collaboration between Product, Engineering, Operations, and other stakeholders. Effective communication is crucial to keep everyone informed about the deployment process, potential challenges, and progress.
2. Embracing Cloud-Native Solutions: Migrating to Blue-Green technology often involves leveraging cloud-native solutions and services, which offers greater flexibility, scalability, and automation capabilities. Embracing cloud-native practices can lead to substantial improvements in application management and deployment efficiency.
3. Training and Skill Development: The transition to Blue-Green technology requires upskilling or retraining the operations and deployment teams. Ensuring that the teams are proficient in managing the new environment and its tools will contribute to a smoother deployment process.
4. Continuous Improvement: Adopting Blue-Green technology should not be considered as a one-time event. The learning here is that one should continually assess the deployment processes, identify areas for improvement, and iterate on the practices to further enhance efficiency and reliability.
5. Compliance and Security: As with any technology migration, it is critical to ensure the compliance and secrutiy measures are maintained during the transition. One should validate that the Blue-Green environment meets all necessary compliance standards and security requirements.

By learning from these lessons and continuously refining the practices, we can successfully make the shift to Blue-Green technology and unlock the benefits of reduced downtime, improved reliability, and enhanced user experiences.

Conclusion

By migrating to a cloud-native infrastructure with Blue-Green Deployment support and leveraging Kubernetes services on Azure, we were able to successfully transform the application release process. The implementation enabled us to achieve higher availability, scalability, and agility, while also improving the overall reliability and performance of the implementation.

References

1. Fowler, M. (2010) Bliki: Blue green deployment, martinfowler.com. Available at: https://martinfowler.com/bliki/BlueGreenDeployment.html (Accessed: 16 January 2024).
2. Fowler, M. (2013) Bliki: Continuous delivery, martinfowler.com. Available at: https://martinfowler.com/bliki/ContinuousDelivery.html (Accessed: 16 January 2024).
3. Hodgson, P. (2017) Feature toggles (aka feature flags), martinfowler.com. Available at: https://martinfowler.com/articles/feature-toggles.html (Accessed: 16 January 2024).
4. Portman, D.G. (2020) Use four keys metrics like change failure rate to measure your devops performance | google cloud blog, Google. Available at: https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance (Accessed: 16 January 2024).
5. Sato, D. (2014) Bliki: Canary release, martinfowler.com. Available at: https://martinfowler.com/bliki/CanaryRelease.html (Accessed: 16 January 2024).