Auto Scaling Jenkins Builders with Packer, Ansible and AWS

At ReadyTalk we’ve recently run into the issue of not having enough builders in our Jenkins instance. We were seeing backups in the build queue as large as 40+ jobs, but commonly we’d see around 20 builds waiting for a build machine. For the most part, those 20 or so jobs were in line to be run on 6 build machines, resulting in hours of backed up builds. Not ideal.

Our old Jenkins build infrastructure

Common build queue

…as an engineer on the ReadyTalk DevTools team it was my responsibility to get serious about our build infrastructure.

Our dev teams were growing more and more impatient with the lack of build machines, and as an engineer on the ReadyTalk DevTools team it was my responsibility to get serious about our build infrastructure. When I started building new build infrastructure I knew I wanted everything to be in code. That’s where Packer and Ansible come in.

Packer. If you don’t already know, Packer is a tool made by Hashicorp for the purpose of provisioning operating system images. They support most, if not all, virtualization software and cloud providers. In my case, I’ll be using Packer to provision AMIs (Amazon Machine Images) in AWS. While Packer does have some provisioning modules (e.g. remote shell), it’s not really meant to be the end all for provisioning an entire OS image. That’s where Ansible comes into play. I’m using Ansible to do all the heavy lifting once the base AMI is built. I’m using Ansible to copy files to my newly minted AMI, make new users, make directories, and install the needed OS packages (apt packages in my case). Packer has support for Ansible, along with other provisioners (Puppet, Chef, etc…), so it was sort of a no brainer for me to use it.

Alright, let’s build the AMI.

First I just wanted to get the AMI building out in EC2, so I started with a very basic Packer JSON file.

{
  "type": "amazon-ebs",
  "access_key": "YOUR KEY HERE",
  "secret_key": "YOUR SECRET KEY HERE",
  "region": "us-east-1",
  "source_ami": "ami-116d857a",
  "instance_type": "t2.micro",
  "ssh_username": "admin",
  "ami_name": "basic_jenkins_builder_{{timestamp}}"
}

Once the file was built all I needed to do was run packer build ami.json and it started spinning up an EC2 instance in my region.

Now that I had a handle on how packer was actually working, it was time to really start building out my AMI with the addition of Ansible. If we want to use Ansible to provision our AMI we’re first going to need to install Ansible on our new AMI. Packer handles that for us using the shell provisioner :)

"provisioners": [  
  {
    "type": "shell",
    "inline": [
      "sudo apt-get install -y python-dev python-pip",
      "sudo pip install ansible"
    ]
  }
]

Now that Ansible is installed, we can actually call it as a provisioner from the Packer code (this would also be in the provisioners section of the Packer file):

{
  "type": "ansible",
  "playbook_file": "./playbook.yml"
}

Once we’ve called our Ansible provisioner, our possibilities are endless. Okay maybe not endless, but there are a ton of Ansible modules.

Now let’s hook up Jenkins to AWS and have it use our new AMI.

First, we’ll need to install the AWS EC2 plugin for Jenkins. Once that’s installed it’s time to configure the plugin (under the Manage Jenkins section of Jenkins). Find the drop down menu labeled Add a new cloud and select the Amazon EC2 option:

Add new EC2 builder

We should now see a new Amazon EC2 cloud entry in your main settings. From here it’s pretty self explanatory to get our AWS account hooked into Jenkins, but there are a few things we should do/pay attention to:

• Make sure we select the correct region for your AMI. AMI IDs are specific to regions. If we have the wrong region, Jenkins won’t be able to find the AMI.
• Once we’ve added the main EC2 information, hit the “Test Connection” button to make sure we can talk to our AWS account. This will save us some debugging headache down the road. Trust me.
• In the AMI section, make sure to hit the “Check AMI” button once we’ve added all the AMI information we see fit. This will make sure Jenkins can actually find the AMI in our given Region and Availability Zone.
• Idle termination time: This will tell our new instance how long to stay up for before tearing itself down after a build. I think EC2 instances are charged by the hour, so 60 minutes seems like the best “bang for our buck” scenario.
• Under the AMI section, there’s an advanced button. Most people won’t care about most of the settings there, but there’s one setting that’s kind of important. It called Instance Cap. This is how we can set a limit on how many EC2 instances we are willing to spin up at any given time. It’s probably best to put a limit on this so we don’t accidentally spin up 100 instances.

Save all our settings, and let’s go test out our new builder. We don’t have to run a job against the new Jenkins AWS builder to test it. We can simply head to “Manage Jenkins -> Manage Nodes” and we’ll see a new button in our node list that says “Provision via <your ec2 instance name>”.

Click on that button and Jenkins will attempt to launch our new instance!

Now we can start assigning jobs to that instance via a label or just let any job run against it.

What does this mean for our build infrastructure?

• Now we’re no longer reliant on static hardware to run builds. Our build infrastructure is now dynamic. Yay!
• The AWS EC2 plugin for Jenkins will “auto-scale”. Let’s say you’ve got one AWS EC2 instance spun up and waiting for a build in Jenkins. A build kicks off on that instance. Now another build gets registered in the Jenkins queue. If the one and only EC2 instance is busy, Jenkins will then spin up another for the build waiting in the queue. No more long waiting queues for builds.
• Most importantly, all of your build infrastructure is now in code. No more hand managed systems. Packer and Ansible now control everything. This means tweaking or creating new AMIs is as simple as some code changes and a code review!

Question about how/why?

Please feel free to reach out to me via Twitter. I kept this article at a high level on purpose, so if you’d like to get into details about exactly how I did something or why I chose this option over another solution, please contact me and I’d be happy to share :)