Orchestrating blue-green deployments with Kubernetes

In this article we’re going to look at a simple example of how to orchestrate a blue-green deployment on a Kubernetes cluster. This article assumes you already have some basic familiarity with Kubernetes and its concepts.

How Kubernetes carries out deployments

Kubernetes currently supports two deployment strategies: RollingUpdate, and Recreate. This can be configured using the strategy field in your deployment spec, and will default to RollingUpdate.

RollingUpdate
The RollingUpdate strategy will replace each pod in the deployment with a new one in turn, such that no outage will occur. This is great if you have developed your application with backwards compatibility so that new deployments can run alongside old ones.

Recreate
The Recreate strategy simply kills all of the existing pods before they are replaced with a set of new ones. This is useful if you have introduced breaking changes which could cause problems if there are pods running your old code and new code simultaneously. The downside here is that there will be a brief outage between the old pods being killed and the new pods being scheduled.

What is a blue-green deployment?

A blue-green deployment is when you run two complete deployments of your application. A deployment that is currently in production, and a deployment with the changes you want to introduce. You then put a proxy or load balancer in front of your application and use it to route traffic to the new deployment when you are ready to take your changes into production.

There’s a couple of great advantages to doing this. One is that by creating a completely new deployment you can be more certain that nothing is broken before you bring your changes to into production. The other is that if something does go wrong which automated testing and health checks didn’t catch, you can quickly and easily switch back because the previous deployment is still running.

The downside is that you need twice as many compute resources as your application actually needs. So if your application requires two machines, you need four machines when you are running both deployments.

You can read more about blue green deployments here.

Orchestrating a blue-green deployment

Using Kubernetes we can easily do blue-green releases with a Service resource to act as our router, and two Deployment resources. We can then toggle the Label Selector on the Service to switch between blue and green.

Firstly, let’s create a Service configuration to act as the router to our blue and green deployments.

apiVersion: v1
kind: Service
metadata:
  name: teapot
spec:
  type: ClusterIP
  selector:
    app: teapot
    deployment: ${DEPLOYMENT} 
  ports:
  - containerPort: 80
    port: 80

Here, I have used a variable called ${DEPLOYMENT} to set the deployment label selector, so that we can easily template between blue and green when we deploy. Initially we will want to point to the blue deployment so let’s apply that change. (Note that you could just use editor patch to make changes to the service, but templating the file and using kubectl apply is easier for checking into version control)

DEPLOYMENT=blue \
envsubst < service.yml | kubectl apply -f -

Now let’s create a Deployment configuration to describe the deployment of our application.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: teapot-${DEPLOYMENT}
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: teapot
        deployment: ${DEPLOYMENT}
    spec:
      containers:
      - name: teapot
        image: teapot:${IMAGE_TAG}
        ports:
        - containerPort: 80
        readinessProbe:
           tcpSocket:
              port: 80
        initialDelaySeconds: 30
        periodSeconds: 5

Here I have used variables for ${DEPLOYMENT} and ${IMAGE_TAG} so that we can quickly template in the docker image tag for our next deployment, and switch between blue and green. It’s important that you define a readinessProbe for your container as we will use this to determine that new deployments of your application have reached a ready state.

Now let’s create our blue deployment:

DEPLOYMENT=blue IMAGE_TAG=v1.0.0 \
envsubst < deployment.yml | kubectl apply -f -

And wait for it to reach a ready state:

kubectl rollout status deployment teapot-blue

Now that these two resources are in place doing blue-green deployments is easy. Simply create the next deployment:

DEPLOYMENT=green IMAGE_TAG=v2.0.0 \
envsubst < deployment.yml | kubectl apply -f -

Wait for it to become ready:

kubectl rollout status deployment teapot-green

And update the selector on the Service:

DEPLOYMENT=green \
envsubst < service.yml | kubectl apply -f -

Now you can interchange between blue and green each time you deploy, and if something has gone wrong at the application level you can easily switch back to the previous deployment by changing the label selector on the Service.

Hopefully this gives you a basic idea of how you can do a blue-green deployment on Kubernetes. There are however a great many other considerations you would want to take when creating a production deployment on Kubernetes which are out of the scope of this article.

You can (and should!) also easily automate this process with a script or your favourite CI/CD pipeline tool.