How I’ve created scaled and distributed Jenkins top of Kubernetes?

Hi everyone,

Everyone should hear the “JENKINS” keyword at least one time in his or her life. Basically, Jenkins is a Continuous Integration server.

To get more detail about Jenkins, please follow that Quora post: https://www.quora.com/What-is-Jenkins-When-and-why-is-it-used

Although Jenkins is one of my favorite CI/CD tool, it has an issue about the capacity management at first. It’s fairly typical to start out with a single master instance. Over time the number and size of Jenkins jobs increase placing more and more demand on the server. Mostly, peoples fix this issue by scaling the Jenkins vertically. But vertically scaling comes with the below problems;

  • Cost per unit of scale increases with the size of the hardware.
  • Complex software configuration required to support a large variety of job types.
  • Increased OPS costs (Server maintaining etc)

Alternatives to vertically scaling;

  • Deploy statically provisioned Jenkins slaves
  • Deploy dynamically provisioned Jenkins slaves.
  • Deploy multiple Jenkins masters allocated; per Environment, Per Business Unit etc.

As you think, we will choose the “Deploy dynamically provisioned Jenkins slaves.” because it requires less effort both before setting up and after set up. Jenkins MASTER will create new slaves before running the jobs dynamically, so we won’t press any button to automate slaves.

So let’s start…

1. Jenkins Service

First thing, we will set up the Jenkins Services. You can create a file called “jenkins-service.yml” and paste the below contents in it.

##########################
#
# This template aims to Orchestrate / Provision Jenkins Services
#
# @author Muhammet Arslan <muhammet.arsln@gmail.com>
# @package tools
# @version 1.0
#
##########################
# [START jenkins_service_ui]
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: jenkins
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
name: ui
selector:
app: master
type: NodePort
# [END jenkins_service_ui]
---
# [START jenkins_service_discovery]
apiVersion: v1
kind: Service
metadata:
name: jenkins-discovery
namespace: jenkins
spec:
selector:
app: master
ports:
- protocol: TCP
port: 50000
targetPort: 50000
name: slaves
# [END jenkins_service_discovery]

To make Jenkins accessible outside the Kubernetes cluster the Pod needs to be exposed as a Service. With a local deployment, this means creating a NodePort service type. A NodePort service type exposes a service on a port on each node in the cluster. It’s then possible to access the service given the Node IP address and the service nodePort.

On the second part of the service template, we will create a ClusterIP-typed ( It’s default type if it’s not typed) to expose 50000 port to all pods in the cluster.

Then create the services in the cluster with above template

kubectl create -f jenkins-deployment.yml

The output should be like;

service "jenkins" created
service "jenkins-discovery" created

To get more details about Service Types: https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

2. Dockerfile

After the services, we need to create the Jenkins Deployment. But before of that, I’ll create a basic Docker image to install all required plugins before the installation.

Create a “Dockerfile” file and paste the below contents in it.

from jenkins/jenkins:lts
# Distributed Builds plugins
RUN /usr/local/bin/install-plugins.sh ssh-slaves
# Scaling
RUN /usr/local/bin/install-plugins.sh kubernetes
# install Maven
USER jenkins

Then build the template

docker build -t geass/jenkins:1.0.2 .

Then tag the Image

docker tag [IMAGE_ID] geass/jenkins:1.0.2

And push to HUB

docker push [IMAGE_ID] geass/jenkins

To get more detail about pushing/pulling images to Docker Hub, please follow that link: https://docs.docker.com/docker-cloud/builds/push-images/

3. Jenkins Deployment

Okay, we are ready to create deployment app for Jenkins. Create a file called “jenkins-deployment.yml” and paste the below contents

##########################
#
# This template aims to Orchestrate / Provision Jenkins Deployment
#
# @author Muhammet Arslan <muhammet.arsln@gmail.com>
# @package tools
# @version 1.0
#
##########################
# [START jenkins_deployment]
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
spec:
replicas: 1
template:
metadata:
labels:
app: master
spec:
containers:
- image: geass/jenkins:1.0.2
name: jenkins
ports:
- containerPort: 8080
name: http-port
- containerPort: 50000
name: jnlp-port
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
# [END jenkins_deployment]

Indicating that, we will use our Docker image from the hub and exposing the both UI and Jenkins Slave Ports to the services. And we will set a JAVA_OPTS parameters to skip Jenkins Setup Wizard, which is enabled by default.

Now create the deployment;

kubectl create -f jenkins-deployment.yml

The output should be like;

deployment.extensions "jenkins" created

4. Check the pods

To be sure, Jenkins Master is running, let’s check the pods.

kubectl get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-85bb8dcc5-d4lgv 1/1 Running 0 1m

Okay, let’s open the service.

minikube service jenkins --namespace jenkins
Opening kubernetes service jenkins/jenkins in default browser...

And boom! Jenkins is ready!

The language might differ on your browser :)

5. Enabling the Jenkins Slaves

With the Kubernetes plugin installed it must be configured by navigating to Manage Jenkins > Configure System and scrolling to the Cloud section. First, we configure the Kubernetes Section as below:

- Click “Add New Cloud” on drop-down button and select “kubernetes” option.

To obtain the Kubernetes URL you should invoke:

$ kubectl cluster-info | grep master
Kubernetes master is running at https:https://www.linkedin.com/redir/invalid-link-page?url=%2F%2F192%2e168%2e99%2e110%3A8443

To obtain the Jenkins URL, first, need to obtain the full pod name of the Jenkins master

$ kubectl get pods --namespace jenkins | grep ^jenkins
jenkins-85bb8dcc5-d4lgv 1/1 Running 0 7m

then obtain the IP address of the pod:

$ kubectl describe pod jenkins-85bb8dcc5-d4lgv --namespace jenkins | grep IP:
IP: https://www.linkedin.com/redir/invalid-link-page?url=172%2e17%2e0%2e14

With these configuration entries, the Jenkins Kubernetes plugin can interact with the Kubernetes API server. Next, we need to configure the pod template and container for the slave so that the plugin can provision a pod. Scroll down a bit, and click “Kubernetes pod template” option on the “Add Pod Template” drop-down.

Then click “Container Template” option in “Add Container” drop-down menu. And fill as below;

Okay! Apply and Save! Configurations are ready! Let’s create a Job.

6. Create a Job!

Click “New Item” button and simply give a name then choose “Freestyle Project” and go ahead.

You will need to set the Label Expression field to match that specified in the Pod Template configuration.

I’ve also created an Execute shell build step.

You are now ready to build the job. Before doing that you should watch the Kubernetes Pods. Do this by installing watch (brew install watch) and executing the watch as follows:

Every 2.0s: kubectl get pods --namespace jenkins                                                                                                                                                                                                                                                                              TRSEUISTMAC028.local: Tue Apr 24 10:08:46 2018
NAME                      READY     STATUS    RESTARTS   AGE
jenkins-85bb8dcc5-d4lgv 1/1 Running 0 29m
jenkins-slave-4svzj 2/2 Running 0 1m

Yes! That’s all! Now your job is running a completely fresh Kubernetes Pod, called “jenkins-slave-*” and after your job finished, your pod will be removed automaticaly.

You can find all the files on my github repo.

https://github.com/geass/jenkins-slaves-on-k8s

Hope to meet your requirements,

Muhammet Arslan

Sr. Devops Engineer