Scaling Jenkins on Demand

This blog explains how to scale up your Jenkins on Demand on AWS and issues encountered while trying to create this Solution.

Editor’s warning: This post is long, but worth it!

On your Jenkins Instance Navigate to plugins Section and Install Following Plugin.

By Navigating to Available Tab.

Search AMAZON EC2 Plugin .

In Below Example I have already Installed the plugin that’s why it’s showing under Installed Tab.

After the installation restart the Jenkins so as that plugin changes are reflected. Navigate to the configure System as show.

To check the EC2 Plugin in the Cloud Section as shown Below.

Now before Proceeding further go back to AWS IAM ROLE and create a user and assign the following roles as shown as we will be using those launch ec2 instance whenever the load increase on the system.

Search Iam as shown in Search bar of Services of AWS as shown.

Click on Roles of AWS as shown. Click on create Role

Click on EC2 service and click on next permission the button unfreezes after selecting the option.

Click on create policy screen as shown. It opens a new tab on the chrome bar.

The policy we used for this plugin.

{
“Version”: “2012–10–17”,
“Statement”: [ {
“Sid”: “VisualEditor0”,
“Effect”: “Allow”,
“Action”: [
“ec2:DescribeInstances”,
“ec2:TerminateInstances”,
“ec2:RequestSpotInstances”,
“ec2:DeleteTags”,
“ec2:CreateTags”,
“ec2:DescribeRegions”,
“ec2:RunInstances”,
“ec2:DescribeSpotInstanceRequests”,
“ec2:StopInstances”,
“ec2:DescribeSecurityGroups”,
“ec2:GetConsoleOutput”,
“ec2:DescribeSpotPriceHistory”,
“ec2:DescribeImages”,
“ec2:CancelSpotInstanceRequests”,
“iam:PassRole”,
“ec2:StartInstances”,
“ec2:DescribeAvailabilityZones”,
“ec2:DescribeSubnets”,
“ec2:DescribeKeyPairs”
],
“Resource”: “*”
}
]
}

Paste in the following policy as shown and click on review policy.

Click on create policy button.

Add user and map this policy with user as shown.

Add the username and choose Access type as programmatic access as shown and click on Next Permissions.

Click on attach existing policy and search of Jenkins policy which we create as of now.

Add the key tags to be used for after wards.

Review the user before creating and everything fine click on create user option.

Download the csv file containing the access_key and secret_key which we will use in the Jenkins console.

Get the ACESS_KEY AND SECRET_KEY which will be used. In below e.g. I have generated

ACESS_Key: XXXXXXXXXXXXXXXXXXXXXXX
SECRET_KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Keep the keypair which we will use to authorize the instance as well.

Once we have all that navigate from Jenkin Home Page

Jenkins-> Manage Jenkins -> Configure System and Go to the EC2 Section right at the bottom and fill in the details as shown and test everything is working fine.

After getting the success message let’s configure the further details. Click on Advanced Section.

Click on Advance Section Pops-Up other option as well.

In Instance Cap base on your usage select the number of instance which you may require to handle the load.

Fill in the details as shown while rolling down.

Click on advance section below the Init script

In this case I have used the same AMI which I used to launch Jenkins Master.

i.e. ami-01e24be29428c15b2 and check ami button to check AMI/ID.

Define instance Type: T1Micro

Availblity Zone: us-west2c

Security group: sg-04e4b787133c6ea5f (The same security groups where Jenkins Master is )

Remote FS root: /var/Jenkins

AMI type: unix

Remote ssh port: 22

Labels: Jenkins_slave

Choose above option from usage drop down box.

Usage: Use this node as much as possible.

Idle termination time: 30

This script will install java which is perquisites for Jenkins slave.

User Data

#!/bin/sh

wget — no-cookies — no-check-certificate — header “Cookie: gpw_e24=http%3a%2F%2Fwww.oracle.com%2Ftechnetwork%2Fjava%2Fjavase%2Fdownloads%2Fjdk8-downloads-2133151.html; oraclelicense=accept-securebackup-cookie;” “https://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.rpm"

yum install -y git

useradd -m -d /home/jenkins jenkins

Now the details provide above will suffice as of now but we can go to see further options as well.

We will leave them black and click on Apply and Save button.

Note: In user Data script you can add the binaries depending what are required for compiling your projects. i.e. maven, ant etc.

Now do some settings at your job so that it can use the Jenkins slave.

I Have selected the free-style project for demo purpose and restrict it to run with the label.

When you type in the label name which we used while configuring the Jenkins Salve it automatically shows up the in the option.

Here I have defined one of my repository it will just checkout the repo

Will add in the required credentials to download the repo.

Just putting echo Hello in execute shell command.

Click apply on button and Save button and run the job and see if it launches the agent.

Before running the job will take the screen-shot of our aws console.

As you can see we have only one instance in us-west-2 zone.

Start the job you can see the pending state.

Let’s see the logs of that job to see what’s happening in the background.

Watch ec2 console to see if it triggers any slave.

Nothing yet we wait for few minutes to see.

Checking AWS Cloud trail to see if it triggers any thing.

Nothing in here Check Jenkins logs to debug.

Something is wrong with the labels

Ok by looking at the log we found following issue.

Exception during provisioning

com.amazonaws.services.ec2.model.AmazonEC2Exception: Value () for parameter groupId is invalid. The value cannot be empty (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: d444f70a-d31e-459a-8de5–316165a92187)

Google it found the following issue on following url.

https://issues.jenkins-ci.org/browse/JENKINS-39960

Need to provide in the subnet-id in the configuration option of ec2 settings.

Lets put in the some more details as suggested in the ticket.

Putting the subnet id as suggested as shown.

Save the settings and lets give it fresh try.

Check the logs this time

We can see it is provisioning a slave instance in the background.

Now we can see some activities on our cloud trail console. It shows that user jekinsclae is creating instances.

Now let see ec2-console.

It has provisioned a Jenkins slave instance.

Let’s have a look to Jenkins job

Still no success.

Let’s have a look to the Jenkins logs

The master machine is unable to connect to the slave.

Ok lets have a look to security groups. We might need to white-list the security groups as well so that machine internally can do ssh internally when they are in same group.

Let’s give it a try now.

Seems to still be problem unable to connect to slave and launch the Job

Seems to be problem with the plugin.

https://issues.jenkins-ci.org/browse/JENKINS-49312

Let’s try with different instance type looks like t1.Micro is too small.

Going with t2samll

It’s a success as of now attaching the Jenkins.log as well.

[root@ip-172–31–40–52 jenkins]# tail -f jenkins.log
Jan 13, 2019 12:16:37 PM hudson.plugins.ec2.EC2Cloud log
INFO: Waiting for SSH to come up. Sleeping 5.
Jan 13, 2019 12:16:42 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to ec2–34–215–30–73.us-west-2.compute.amazonaws.com on port 22, with timeout 10000.
Jan 13, 2019 12:16:52 PM hudson.plugins.ec2.EC2Cloud log
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jan 13, 2019 12:16:52 PM hudson.plugins.ec2.EC2Cloud log
INFO: Waiting for SSH to come up. Sleeping 5.
Jan 13, 2019 12:16:57 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to ec2–34–215–30–73.us-west-2.compute.amazonaws.com on port 22, with timeout 10000.
Jan 13, 2019 12:17:07 PM hudson.plugins.ec2.EC2Cloud log
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jan 13, 2019 12:17:07 PM hudson.plugins.ec2.EC2Cloud log
INFO: Waiting for SSH to come up. Sleeping 5.
Jan 13, 2019 12:17:12 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to ec2–34–215–30–73.us-west-2.compute.amazonaws.com on port 22, with timeout 10000.
Jan 13, 2019 12:17:22 PM hudson.plugins.ec2.EC2Cloud log
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jan 13, 2019 12:17:22 PM hudson.plugins.ec2.EC2Cloud log
INFO: Waiting for SSH to come up. Sleeping 5.
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to ec2–34–215–30–73.us-west-2.compute.amazonaws.com on port 22, with timeout 10000.
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connected via SSH.
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: connect fresh as root
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to ec2–34–215–30–73.us-west-2.compute.amazonaws.com on port 22, with timeout 10000.
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Connected via SSH.
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Creating tmp directory (/tmp) if it does not exist
Jan 13, 2019 12:17:27 PM hudson.plugins.ec2.EC2Cloud log
INFO: Verifying: java -fullversion
Jan 13, 2019 12:17:28 PM hudson.plugins.ec2.EC2Cloud log
INFO: Verifying: which scp
Jan 13, 2019 12:17:28 PM hudson.plugins.ec2.EC2Cloud log
INFO: Copying slave.jar to: /tmp
Jan 13, 2019 12:17:28 PM hudson.plugins.ec2.EC2Cloud log
INFO: Launching slave agent (via SSH client process): ssh -o StrictHostKeyChecking=no -i /tmp/ec2_8286952618637243926.pem ec2-user@ec2–34–215–30–73.us-west-2.compute.amazonaws.com -p 22 java -jar /tmp/slave.jar
Jan 13, 2019 12:17:40 PM hudson.slaves.CommandLauncher launch
INFO: agent launched for Amazon Ami (i-050c643bef607384b)
Jan 13, 2019 12:17:44 PM hudson.model.Run execute
INFO: test #5 main build action completed: SUCCESS

Summary: The plugin dynamically provisioned a Jenkins slave on ec2-instance as required for the needs and shuts it down automatically if no job is running. You can also save the cost by reducing the idle timeout as well as using spot instances.

Based upon the load and threshold you can keep a tap on number of nodes required.