Quick and Simple — How to Setup Jenkins Distributed (Master-Slave) Build on Kubernetes
Even today, Jenkins still bring massive improvements to the open source community, especially under the DevOps field of view. Its highly customisable features and growing community has been helping developers of many stacks around the world. For a small company a daily build can be handled with ease, but for a big and busy company, build time could be resource consuming and highly strenuous which makes the SDLC process highly susceptible for delay on delivery caused by snowball effect.
Kubernetes also becoming a major revolutionary tool for deployment strategy which offers highly scalable and stable deploy environment that could become a solution for the scalability problems that Jenkins faced. Within this tutorial, we will combine both powers and achieve greater scalability for Jenkins using Distributed Build or Master-Slave strategy.
Without further ado, let’s begin!!
Prerequisites
- A working Kubernetes cluster, for this tutorial I’m using minikube.
- Kubernetes CLI (kubectl)
- I’m using jenkins (v2.237) image and jenkins slave (v4.3–1) image from official jenkins docker hub.
1. Setup Jenkins Deployment
You can deploy Jenkins at any namespace, but for better isolation it is recommended that you create a dedicated namespace.
kubectl create namespace jenkins
Create a Jenkins deployment that will assume role as master node, named jenkins-master.
# deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins-master
namespace: jenkins
spec:
selector:
matchLabels:
app: jenkins-master
replicas: 1
template:
metadata:
labels:
app: jenkins-master
spec:
containers:
- name: jenkins
image: jenkins/jenkins
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
- name: http-port
containerPort: 8080
- name: jnlp-port
containerPort: 50000
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
emptyDir: {}Here you need to open 2 ports, which are
8080 : will be used for accessing Jenkins API
50000 : will be used by Jenkins slave nodes (jnlp) and master node to communicate with each other.
You can configure above setting with another port, but you need to reconfigure Jenkins for that purpose.
Deploy Jenkins using kubectl:
kubectl apply -f deployment.yaml
2. Add Kubernetes Service Account and Role for Jenkins
Jenkins needs to access the Kubernetes API, therefore you need to properly setup a Kubernetes Service Account and Role in order to represent Jenkins access for the Kubernetes API.
Create jenkins-master service account:
kubectl create serviceaccount jenkins-master -n jenkins
Create required access Role and RoleBinding for jenkins-master :
# role.yaml---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: jenkins
name: jenkins-master
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins-master
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins-master
subjects:
- kind: ServiceAccount
name: jenkins-master
Apply the Role and RoleBinding manifest:
kubectl apply -f role.yaml
3. Add a Kubernetes Service to Access Jenkins
You need expose jenkins-master pod to the network in order to setup Jenkins properly, to enable access from network to the Jenkins pod you need to create a Kubernetes Service for jenkins-master .
# service.yaml---
apiVersion: v1
kind: Service
metadata:
name: jenkins-master
namespace: jenkins
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
name: jenkins-master
- port: 50000
targetPort: 50000
name: jenkins-jnlp
selector:
app: jenkins-master
Apply the Service manifest:
kubectl apply -f service.yaml
To access Jenkins UI, you can use kubectl port-forward by forwarding to port 8080 of the jenkins-master pod.
kubectl get pod -n jenkins , to view the jenkins-master pod name (e.g. jenkins-master-5f8c97b8-wwdqz)
Then use port-forward command to access Jenkins through port 8080:
kubectl port-forward -n jenkins jenkins-master-5f8c97b8-wwdqz 8080:8080
Now, you can access Jenkins UI via http://localhost:8080
4. Install Required Plugins on Jenkins
Install required plugin to support distributed build setup:
- Kubernetes Plugin
- Docker Plugin
- Git Plugin
- Node and Label Parameter Plugin
*For this tutorial, I will be using an NodeJS project so I also need the nvm-wrapper plugin.
To install plugins navigate to Manage Jenkins > Manage Plugins > Available
5. Setup Jenkins Cloud Configuration and Slave Pod Specification
First, you have to register the jenkins-master service account to Jenkins credential manager, navigate to Credentials > Systems > Global Credentials (or you can add your own domain).
In this tutorial I registered 2 credentials:
- My SSH private key for Github credential (~/.ssh/id_rsa)
jenkins-masterservice account token:kubectl get secret $(kubectl get sa jenkins-master -n jenkins -o jsonpath={.secrets[0].name}) -n jenkins -o jsonpath={.data.token} | base64 --decode
Next, you have to get these values to setup cloud configuration properly:
- Kubernetes API server address:
kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " " - Kubernetes server CA certificate key:
kubectl get secret $(kubectl get sa jenkins-master -n jenkins -o jsonpath={.secrets[0].name}) -n jenkins -o jsonpath={.data.'ca\.crt'} | base64 --decode
To configure cloud provider navigate to Manage Jenkins > Manage Nodes and Clouds > Configure Clouds.
Important: use kube DNS name (http://jenkins-master.jenkins.svc.cluster.local) as Jenkins URL instead of http://localhost:8080 or http://127.0.0.1:8080. And test connection to make sure everything works. You also have to configure
“Jenkins URL” field on Manage Jenkins > Configure System with the same value.
Important: Here’s one of the cool features, you can limit resources that the pod will use just as you limit a Deployment.
You have to label the pod as jenkins-slave in order to restrict the build only on slave nodes.
6. Set Up a Project
For this tutorial, I will be using my own NodeJS project called url-shortener (https://github.com/antoooks/url-shortener).
You need to make sure that:
- The project only execute on slave node with correct label which is
jenkins-slave.
- The project have access to target git repository. I’m using SSH key registered at Jenkins credential manager
- The project has correct environment setup. In this case, I’m using the NVM wrapper which will provision Node and NPM with desired version on the vanilla slave node image. Usually, you need to provision your own build slave image based on Jenkins slave vanilla image, but in this tutorial, NVM wrapper is all I need, so I don’t need to rebuild slave image.
- You’ve defined build steps for the project correctly. In this case, my build step is simply
npm run build.
7. Start Building!
Go to your project and hit “Build Now”, if you done everything correctly you will see that Jenkins is provisioning your slave node inside a Kubernetes pod when you navigate to “Console Ouput”.
You can also see that the build job is running on labeled slave node in your Build History sidebar inside your project page.
If you still failing to spawn slave node, you can view event logs in Manage Jenkins > System Log > All Jenkins Log to help you get clearer messages.
And that’s a wrap! I hope you enjoy this article. If you have any question, just drop your comments below and don’t forget to follow this account for more articles like this.
Thank you and good luck.
References:
** Disclaimer: this article hasn’t been proof read because I don’t have an editor to do it. 😔
