Blue/Green Rails app deployments in Google Container Engine (GKE)

8 min readAug 9, 2017

In a previous article, I had described the process of deploying a Rails App to Google Container Engine and how to use Kubernetes to achieve this.

In this article, I will focus on various application deployment strategies in GKE and any impact that these deployments might have on our customers. We will also see how to use Blue/Green Deployments (Option 3 below) to alleviate some of the concerns with downtime during deployments.

While the sample code refers to a Rails application, these principles can be applied to applications built in any language.

Please read the first article to ensure that you have a working knowledge of GKE and Kubernetes and that you have created a Google Cloud cluster that will be used in the sections below.

Motivation:

Let’s take an example of a web application that is currently live (deployed to Production) and is serving multiple customers. In Article 1, we deployed a sample Rails app successfully to GKE. Let’s call it version v1 of our app.

Now, let’s say that there’s a new version of this application (v2) with even more features. How do we deploy this to GKE without impacting our customers?

Our user traffic could range from very small (a few customers a day) to millions of customers a day (think Google, Amazon, PayPal etc.). While a new application might start small, it would still need to be built with scalability in mind (both from a feature and a user base perspective).

If built right, there is a large, scalable, very successful application within every small app :)

With an application’s popularity comes a larger user base and a somewhat disproportionate impact if there’s any downtime (along with bad publicity).
How we get to the perfect app might be beyond the scope of this article, but how we can improve our end users’ experience while we get there is what this is all about.

So, what are the various ways of deploying our application to GKE?

Option 1: “Clean” deployments: We could do a new deployment with version v2 (and every subsequent version) of our app using Kubernetes. In essence, you are bringing down the old deployment and creating a new one with the latest version. This usually goes like “Let’s pick a time window when few users are on our site and deploy v2.” However, if your application has say, 20 replicas (instances) hosting v1, there will be some downtime when the new container images for v2 are deployed. And, just in case v2 doesn’t turn out to be behaving as expected, you would need to be able to switch back to v1. More downtime. If you own an application that has international users, the time window to make changes gets even smaller.

Option 2: Rolling updates: Rolling updates allow for incremental deployments without having to do full ones as in Option 1. This approach is better than #1 above but still comes with delays that can happen when there are greater numbers of pods that need to be refreshed.

In the first article, I used Kubernetes rolling updates to push incremental changes to GKE.

kubectl set image deployment web web=$IMAGE — record
kubectl rollout status deployment web

Our deployment yml file contains the entries below. Here, the maxSurge would be the incremental number of pods created during the deployment…for example: if we have 2 pods (replicas: 2 below) , we would have 1 more (3 total) and maxUnavailable is the number of pods that would be unavailable during the rollout.

spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1

Rollbacks can be done as follows but will still come with s0me downtime overhead.

# For ROLLING BACK to previous version
kubectl rollout history deployment web
kubectl rollout undo deployment web

Option 3 (Recommended): Use Blue/Green Deployments.

Blue/Green Deployments allow us to deploy 2 versions (the current version and the next) of our application to Production and allow us to switch seamlessly between them. This helps with our goal of achieving Zero Downtime.

Please see Martin Fowler’s excellent explanation on this topic in the link above.

This strategy answers the following questions:

How do I deploy a new version of my application to Production but get to test it first (smoke test) before my customers start accessing it?
How do I switch my customers over to this new version without any downtime? This is called Zero Downtime Deployment
How do I switch back to my last stable version in case something goes wrong with this new version?

This concept is not new. Many organizations have very successfully done this and gone beyond this as well. A sample deployment in GKE is shown below:

Blue/Green Deployments in Google Container Engine

Prerequisite: Create a Google Container Cluster. This cluster will contain Node Pools which are instance groups that run Kubernetes.

The process is as follows:

Part I — initial deployment

Step 1: Engineers build code and check it into a source code repository (GitHub in the above example).

Step 2: CI/CD tools such as CircleCI, Jenkins etc. detect these checkins and build and test the code automatically. If the tests pass, the application is deployed via a Kubernetes Deployment (called Blue Deployment in this case). This deployment has a configuration that sets the number of replicas to create. This step can also be done manually but is a good candidate for a CI/CD tool.

Step 3: A Kubernetes Service (Production Service) is created that will allow access to these containers. This is a one time setup

Step 4: An Ingress (Production Ingress) is created that allows outside users to access our application deployed via the Blue Deployment. This is a one time setup

Part II — subsequent deployments

Step 1: Same as Step 1 above

Step 2: This time, the CI/CD tool would pick up the latest changes, detect the previous deployment (Blue) and deploy the new changes to a Green Deployment

Step 3: A Kubernetes Service (Smoke Test Service) is created that will allow access to these containers. This is a one time setup

Step 4: An Ingress (Smoke Test Ingress) is created that allows internal users to access our application deployed via the Green Deployment. This is a one time setup

Step 5: Once internal users smoke test the Green Deployment and everything’s working well, the Production Service is modified (by changing the keyword “blue” to “green” in the yml file, to point to the Green Deployment. This switch takes a few seconds

Step 6: The end users (customers) are now seeing v2 of the application (Green Deployment).

Step 7 (Optional): If there are any issues with the Green Deployment version, the Production service can be modified to change the “green” to a “blue” in the yml file. This switch takes a few seconds

Step 8: Blue and Green Deployments are used to alternate between subsequent versions

Note:
With this strategy, you will gain the advantage of Zero Downtime but it comes with the additional cost of having 2 sets of deployments that exist in parallel within the same cluster.
Caution: If there is a single, common database used between these version updates, please note that any migrations/seeds (For Rails applications) that are required for the new version, need to be backward compatible. For example, if v2 needs some new columns and later it is deemed to be unstable and you switch back to v1, the database changes should NOT adversely impact v1.
Application of Blue/Green Deployments to databases and other dependent data stores is beyond the scope of this article.

Implementation Details:

Assume you have a website (https://mywebsite.com) that is deployed to Production. Assume the current version is v1 from a Blue Deployment. The public IP address for this website is obtained from the Production Ingress.

As in the diagram above, when you do a Green deployment, you can test this version by going to something like https://smoketest.mywebsite.com. (or whatever IP address is provided by the Smoke Test Ingress)

New files: We will need 2 deployment yml files and 2 Service yml files corresponding to the Blue and Green Deployments. Finally, you would need 2 Ingress files to complete the above picture.

Create a deployment file for Blue and Green deployments each. The metadata name and metadata labels identify the deployments as blue and green.

web-deployment-blue.yml

web-deployment-green.yml

Create the Production Service (web-service-production) that points to the Blue deployment. The selector (type and color) identify the deployment that the service points to.