Creating a Super Available Jenkins Master on AWS

I’m going to write some articles about Jenkins scalability using AWS. The most important thing about scalability when we are talking about Jenkins is set on the slaves side of the architecture.

However, before we start it’s worth mentioning the importance of protecting the Master due to the environment configuration being stored there!. It’s not fancy, but it is very important :)

In this article you will learn a way to set your Jenkins Master as an auto-healing and cost-effective part of your continuous integration and deployment system.

First we need to ensure the Master’s ability to be restored automatically after any problem (or almost any problem), so we are going to set the Master node within an auto scaling group (min and max = 1 instance) and an ELB in front of it.

Second we need to ensure that we don’t lose any amount of configuration information; for example: taking EBS snapshots and saving backups to S3 can work, but then we can lose the new information produced between two backups or two snapshots. We are going to use EFS, to ensure that 100% of the information is saved and stored in a sharable system.

If you have a group of two or more Masters, maybe you should double check this strategy because there can be data issues when all Masters try to write to EFS at the same time.

The final picture is something like this:

First we are going to create a EC2 based on Ubuntu:

You can choose a t2.micro and the correct connectivity options so you can access the instance using SSH (we’ll need to do several things in there). After you finish the creation wizard, while the EC2 is starting, let’s create a EFS File System:

Let’s select the same VPC as the EC2 and the correct Security Groups so the EC2 can connect to the new File System:

Let’s go back to the EC2 instance and let’s connect to it. If you are not sure how to connect to your EC2 using SSH just select your instance and the “Connect” option in the “Actions” menu. Copy and Paste the command and keep an eye on your .pem and the Ubuntu user.

Now that we are connected let’s check the commands to mount our new File System on the new EC2, and to know the commands let’s check the EFS console:

So, there we have the commands but we are going to do something different: we are going to create the directory /var/lib/jenkins because that’s the default Home directory of Jenkins, the place where all the configuration is stored:

sudo apt-get install nfs-common
cd ..
cd ..
cd var
cd lib
sudo mkdir jenkins
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-5640c49f.efs.eu-west-1.amazonaws.com:/ jenkins

We are mounting the EFS File System in that directory, so all the configuration is going to be saved in EFS and can be used by another instance in the case of failure.

Once the File System is mounted we can install Jenkins:

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

And after installation we’ll need to access the EC2 public address using the 8080 port to get this:

Once the configuration wizard is finished installing the required plugins and has created your admin user, we can finally access the Jenkins home page:

Ok! Now it’s time to create a new AMI using the EC2 instance:

Give it a name and…

Done!! We have our base AMI and the EFS File System with all config files!! Now we are going to use it to create our real environment.

Let’s go to Auto Scaling Group in the EC2 console:

Then create a new Launch Configuration, select your instance type, give it a name and in “Advanced Details” use the “User Data”:

#cloud-config
package_upgrade: true
packages:
- nfs-utils
runcmd:
- chown ubuntu:ubuntu /var/lib/jenkins/
- chown jenkins:jenkins /var/lib/jenkins/
- echo "fs-5640c49f.efs.eu-west-1.amazonaws.com:/ /var/lib/jenkins nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab
- mount -a -t nfs4

Maybe you should add some extra grant permissions commands here to the Jenkins user for the var/lib/jenkins directory. If you get a permission denied error at any point, just grant extra permissions and restart the Jenkins service.

What did we do?? We are just mounting the EFS File System on the new launched EC2 instance, because when you created the AMI before, it didn’t save the EFS mounting information.

Now, finish the wizard while keeping an eye again on the connectivity options (Security Groups).

Now create the new Auto Scaling Group and remember to use private subnets because we are going to access Jenkins using the ELB:

Set the min and max instances to 1:

Finish the wizard, now we have the new Auto Scaling Group and a new EC2 instance is going to be created based on the new Launch Configuration.

Finally let’s create a new Application Load Balancer:

Select your name, listener (8080), subnets, security group and a target group selecting the brand new EC2 instance:

Aaaaand Done!!!! Now you have your Super Available Jenkins Master!

You can access your new Jenkins using your ELB DNS:8080.

If you want to test it just go and terminate your EC2 instance, a new one will be launched automatically with the same configuration, no data will be lost and it will be accessible through the same ELB DNS:8080.

Are you just testing? Then visit this article and find an easy and fast way to drop everything you just did and get ready for the next one!