Deploy Jenkins agents using EC2 Spot instances

The purpose of this document is to go through a (Proof of Concept) POC that will help scale Jenkins nodes/jobs cost-effectively.

Inspired by Lyft’s Solution, we are leveraging a plugin offered by Jenkins and AWS Auto Scaling service to meet the demand of the jobs served by the Jenkins master node.

What are we aiming for in this POC?

• We are able to execute jobs in the spot instances
• It scales instances as needed
• Prove that Spot instances are helping to cut costs
• We have confidence to reduce the instance type of master node

Technologies that we will be working with are

High level on how these components will be interacting

How AWS Architecture looks like

Steps to Create this POC

Setup Jenkins Master Node

• Inside your AWS Console, click on EC2 Service
• Then click on Launch Instances
• Select Amazon Linux 2 AMI 64-bit (x86)
• Choose t2.micro, Click Next on Configure Instance Details
• Choose the VPC you wish to deploy this Jenkins instance. Make sure your VPC has DNS hostnames and DNS resolution enabled
• Choose a public subnet inside your VPC that has a route to the internet gateway.
• Enable Auto-Assign Public IP
• In the user data section — paste the following:

#!/bin/bash
wget -q -O — https://pkg.jenkins.io/debian-stable/jenkins.io.ke
sudo sh -c ‘echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list’
sudo yum update -y
sudo yum install java-1.8.0-openjdk -y
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm — import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo amazon-linux-extras install epel -y
sudo yum install jenkins -y
sudo service jenkins start

• Leave others as default, click on Next: Add Storage
• Leave storage default
• Add any tags you like, Click on Next: Add Tags
• Click on Next Configure Security group
• Click on Review and Launch
• Click on Launch after reviewing your information
• Create a key pair that you will use to connect to the instance, select the acknowledge
• Click on Launch Instances

Setup Jenkins

• After the EC2 instance is finished with initialization and all status checks have been passed, get the public IP address of the instance and open it in a browser in a format like this 3.139.62.228:8080 with a port number of 8080. You should be able to see Jenkins welcome page.

• To retrieve this password use this command in your instance:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

• Copy and paste the output to the Jenkins page, click on continue
• Choose Install suggested plugins. Wait till everything is installed
• In the “Create First Admin User” page, create your user that will be used to login to the Jenkins console in the future. After, click on save and continue
• In the Instance Configuration page, acknowledge and leave the defaults. Click on save and finish.
• Click on “Start using Jenkins”

Create an AWS IAM user for EC2-Fleet Plugin

• In your AWS console, head to IAM service to create a new user. This user will be used by the EC2-Fleet plugin (which we will install later) that install spot instances for us.
• In the IAM service, click on Users on the left panel and click on “Add Users” on the right side.
• Type in a username that makes most sense, for example (Ec2-fleet-user).
• In the AWS Access type section, highlight the Access Key — Programmatic Access
• Click Next on permissions
• In the Set Permissions Section, click on “Attach existing policies directly” and click on “Create Policy” to manually create a policy.
• Switch to json view and paste the following permissions

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeSpotFleetInstances",
            "ec2:ModifySpotFleetRequest",
            "ec2:CreateTags",
            "ec2:DescribeRegions",
            "ec2:DescribeInstances",
            "ec2:TerminateInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSpotFleetRequests"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:UpdateAutoScalingGroup"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListInstanceProfiles",
            "iam:ListRoles",
            "iam:PassRole"
         ],
         "Resource":"*"
      }
   ]
}

• Click Next on Tags
• Click Next on Review
• Name the policy that makes most sense, for example (ec2-fleet-permissions-policy)
• Click on Create Policy
• Head back the tab where you were creating the role. Click on the refresh button, click on “Filter Policies” and choose customer managed.
• Choose your policy that you created earlier. If you cant find it, refresh again. Click on “Next: Tags”
• Click on Next: Review
• Click on Create User
• Download the .csv file for later use. Click on Close

Create a launch template and auto scaling group in AWS Console

• A launch template includes the information about the instances that auto scaling group need to launch
• Head to the EC2 service inside your AWS Console. Click on Launch templates on the left panel. Click on Create launch template
• Name the Launch Template for example (EC2-fleet-launch-template)
• In the Launch Template Contents section, choose the amzn2-ami-hvm-2.0…x86
• Choose t3.small for Instance type
• Choose an existing key pair or create a new key pair that will be used to connect to these spot instances
• In the network settings, choose VPC and same security group as your Jenkins Server for now
• Expand Advanced Details, choose “Request Spot Instances”. Inside the user data, paste the following

#!/bin/bash -xe
sudo amazon-linux-extras install epel -y
sudo yum update -y
sudo yum remove java-1.7.0-openjdk
sudo yum install java-1.8.0-openjdk -y
sudo yum install git -y
sudo yum -y update aws-cli

• Click on Create Launch template
• Next inside the same EC2 service, click on Auto Scaling Groups on the left panel
• Click on Auto Scaling Group
• Name the Auto Scaling Group — For example (EC2-Fleet-AutoScaling-group)
• In the Launch template section — choose the template you have created earlier. Click Next
• Choose the VPC where the ec2 instances will be launched
• Choose all the private subnets as the availability zones. If your vpc doesnt have atleast 3 subnets, create them and come back to this menu
• Click on Next
• We will skip the load balancer for now, click on next

Group Size

• Desired capacity = 1
• Minimum capacity = 1
• Maximum Capacity = 4

Skip the scaling policies for now

Click on Next

Skip Notifications for now, click on Next

Skip Tags, Click on Next

Click on Create Auto Scaling Group

You should see one of the instances already spinning up

Install EC2-Fleet Plugin

• Head to Jenkins Dashboard. Click on Manage Jenkins on the left panel
• Click on Manage Plugins
• Choose Available and search for ec2-fleet
• Select the plugin and click on install without restart
• Click on restart Jenkins after installation is complete and no jobs are running
• Log back in with your credentials

Setup EC2-Fleet Plugin

• Go to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds
• Click on the drop down menu “Add a new Cloud”, select Amazon EC2 Fleet
• In the AWS Credentials, click on Add drop down menu
• Change Kind to AWS Credentials
• Change scope to System (Jenkins and nodes only)
• Add the ID that specifies the username that holds these credentials
• Add the Access Key ID and Secret Access from the csv you have downloaded
• Click on Add
• Now click on the drop down for AWS Credentials and choose the credentials you have added
• Change region to your respective one
• EC2 Fleet section should be auto populated with the auto scaling group you have created in your AWS Account
• Click on Test Connection
• In the Launcher Section
• Change Launcher drop down to “Launch Agents via SSH”
• Click on “Add” for Credentials
• Change Kind to SSH Username with Private Key
• Change scope to System (Jenkins and nodes only)
• Add “ec2-user” to ID and Username
• For Private Key, click on Enter Directly
• Paste the contents of your .pem file that was used to create the main jenkins server
• Click on Add
• Choose the new credentials you created
• For Host Key Verification Strategy: Choose Non verifying verification strategy
• Choose Private IP option to be accessed internally
• Change label to “spot-agents”

Add the following parameters:

• Number of executors: 1
• Max Idle Minutes before scaledown: 5
• Minimum Cluster Size: 1
• Maximum Cluster Size: 5
• Maximum Total Uses: -1 (default)

Leave the rest default. Click on Save

Add a sample Jenkins Build

• In the Jenkins Dashboard, click on New Item on the left panel
• Enter an item name “sai-test-build”, choose freestyle project. Click on Ok
• Choose Git
• Add this repository url: https://github.com/psdilip/python.git
• Inside the Build Section — choose Execute Shell and add the following command

python3 basics.py

• Click on Save

Run the Sample Jenkins Build

• In the Jenkins Dashboard, verify that a fleetcloud instance is running inside the “Build Executor Status”
• Click on the play button on your sample project to start the build
• View the output by clicking the project, clicking on the latest build history and on console output

Summary

• We used the EC2 Plugin to deploy spot instances as Jenkins Agents. Our goal was to remove load from the Jenkins Master and have agents scale accordingly to the number of jobs that are invoked parallelly. This is a good start in thinking how to approach scalability in big projects.