Automated AWS AMI builds for Jenkins agents with Packer
Everyone knows that up to date software is good for your health.
We recently noticed that our builds on Jenkins took around 10 minutes to finish vs 1m 30s on our laptops (thanks Felix!). It turns out that someone (clearly not me) was lazy and the Maven dependency cache on the agents was months out of date. So every single build spent 5+ minutes in one of the 9 levels of dependency hell.
To fix this, I decided to build Jenkins images daily and programmatically. This gives us a ‘golden image’ with:
- software updates and OS security patches
- an up to date dependency cache for faster builds
You’ve come to right place if your:
- project’s dependency resolution takes a long time or is unreliable, *cough*npm
- CI pipeline relies on AMI IDs being updated manually in the Jenkins
As you’re reading this post, you must be already using the Amazon EC2 plugin for Jenkins. Otherwise, head over to the CloudBees blog post on how to set up Jenkins agents in the cloud and save yourself some money by using spot instances on demand.
Things you will need:
- a running Jenkins 2.x instance
- the Amazon EC2 plugin installed (v1.29 or later)
- the Matrix Authorization plugin installed
Building with Packer
Packer is an amazing tool by Hashicorp to automate the creation of any type of machine image. You can write your image definition as JSON and make it portable between different backends, be it a cloud provider or virtualisation technology. If you’re deploying multi-cloud solutions with golden images, look no further.
Packer’s strength lies in the support for a wide variety of tools and platforms. It has builder integrations with most popular platforms: Amazon AWS, Google Compute Engine, Microsoft Azure, Docker, VirtualBox, etc. You can also use your favourite infrastructure as code tool to bootstrap your machine.
In this case we will be using the Amazon EC2 AMIs builder and the Ansible Remote provisioner.
Our Packer file looks like this:
One thing I found funny was the peculiarity when specifying the format for isotime — why would someone do that when there’s already a good way to specify the format? Apparently it’s a GoLang thing…
Updating the AMI ID in Jenkins
Set up Jenkins
Packer needs to be able to access the job logs as well as change the AMI in the settings.
Create a packer user in Jenkins and set these permissions if you’re using Matrix-based security:
- on the project (Jenkins job): Job > Read
- global security (Settings): Overall > Administer
IAM permissions in AWS
You need to create a role with permissions for Packer to access/edit EC2 resources; do it in the AWS console IAM section and call it ‘jenkins-ec2-role’. For simplicity, we attach the AmazonEC2FullAccess policy. Note the Instance Profile ARNs at the top when you create the role.
You need to attach this role to agents that will create the AMIs.
In Jenkins Settings, go to Cloud > Amazone EC2 > AMIs; click on ‘Advanced’ and fill out the ‘IAM Instance Profile field:
The magic glue
All the hard work has been done in this PR by Marvin Pinto. Thank you!
There is a Python script attached to the PR that can be run in a post-build step and it will update the AMI ID in the EC2 Cloud of your choice.
My updated version of Python script has a few fixes:
- support for CSRF (on by default for 2.x installs)
- login to obtain the build job logs for private projects
- verifies the website SSL certificate by default
Link to project: https://github.com/burukuru/jenkins-update-ec2-ami
Do the swap
Putting it all together in a Jenkinsfile:
Change the username/password, region and the variables below:
EC2_CLOUD_INSTANCE is the name of the Amazong EC2 cloud in the plugin (as seen in the screenshot below).AMI_PROFILE_NAME is the AMI’s “Description” in the EC2 plugin.
Now that’s automated, I can go on holiday. The mountains are calling.
If you enjoyed this article, please hit the clap button below, so others can find it, too. Please get in touch if you’ve any questions or comments, either below, via our Community Slack channel, or via our discussion forum.
Find out more from https://grakn.ai.
