Blue - Green Deployment with Jenkins

Recently, we have started to use a continuos delivery pipeline and I want to share my experience about our transition from manual deployment ( aka. copy-paste) to continuos delivery.

How Was It Before?

arabam.com applications

We have 6 different applications for now. And 2 of them have multiple nodes. So, if someone needs to deploy new features to live, he or she needs to copy files to every application. It’s really really hard to do.

Eventually, we’ve decided to setup a continuous delivery pipeline though it’s too late.

I will try to explain only web and api deployment in this post.

Blue — Green Deployment

A blue/green deployment is a change management strategy for releasing software code. Blue/green deployments, which may also be referred to as A/B deployments require two identical hardware environments that are configured exactly the same way. While one environment is active and serving end users, the other environment remains idle.
If problems occur after the switch, traffic can be directed back to the idle configuration that still runs the original version.

This is the most important point; If you setup a blue-green deployment pipeline, you will be able to test in live!

http://searchitoperations.techtarget.com/definition/blue-green-deployment

Continuous Delivery

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.
Continuous Delivery doesn’t mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time.

Continuous Delivery vs Continuous Deployment

Source: https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff

We didn’t want to setup a continuous deployment pipeline. We want to control deployment manually.

Tool Options

After deciding continuos delivery, we had a lot of options and we studied most of them. I will give you some options.

Jenkins
Team City
Bamboo
Go
  • Web site: https://www.gocd.org/
  • Open source — ThoughtWorks company
  • Platform Independent — Java Based
  • Server-Agent relation
  • 50–100 Plugins
Travis

After long researches, blue ocean plug-in made it easier for us to choose jenkins.

Jenkins Pipeline and Blue Ocean

Jenkins Pipeline (or simply “Pipeline” with a capital “P”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Jenkinsfile Example:

pipeline {
agent any
    stages {
stage('Build') {
steps {
sh 'make'
}
}
stage('Test'){
steps {
sh 'make check'
junit 'reports/**/*.xml'
}
}
stage('Deploy') {
steps {
sh 'make publish'
}
}
}
}
  • Formerly known as workflow.
  • Distrubutable!
  • Visualized!
  • Single and simple groovy script and no more a lot of job configurations

Our Deployment Pipelines

After describing the terms, I can show you our setup.

We didn’t use Jenkins pipeline source control feature. We created “New Pipeline Item” and configured pipeline script in that.

  • We have 4 pipelines for now.
  • We have email notification at the end of every pipeline.
  • I will give you Blue and Green pipelines. Others are almost same.

Blue Pipeline

This is the starting point.

blue pipeline (sorry for typo(*roting))
  • Deploys api and web site.
  • Pulls branch last revision.
  • Builds and prepares deployment packages
  • Stop Sites(Stop load balancer routing, stop IIS)
  • Copy new files
  • Start Sites(Start only IIS, not routing)

Green Pipeline

If everything is fine in blue servers, we can now go to green pipeline and live.

green pipeline (sorry for typo(*roting))
  • Takes parameter to deploy which version (We prepared packages in blue)
  • Activate Blue Sites Routings(Load balancer)
  • Stop Sites(Stop load balancer routing, stop IIS)
  • Copy new files
  • Start Sites(Start only IIS, not routing)
  • Copy new files
  • Activate Blue Sites Routings(Load balancer)

And we are LIVE! Load balancer now sends requests to all servers (blue and green)

Extra Informations

Deployment Server

We have one dedicated server. It has 4gb ram, Intel Xenon 2 processor CPU and 100gb hdd.

We created a folder and copied and created all necessary things in that.(Batch scripts, status files, source control, PPsExec, Nuget, MsBuild, Created Build Packages in Pipelines)

We have all source files in deployment server. It connects scm with access key. We use bitbucket and mercurial.

Routing

We use F5 for load balancing.

We have abc.txt files in every server. Load balancer reads files at certain intervals. If it sees “OK” in file, sends requests to that server.

We have active and passive files in deployment server. We copy them when necessary(activate or passivate).

IIS Operations

We used PsExec.exe to start-stop-recycle site and application pool.

Batch Scripts

I created a repo in github. You can find all scripts there.