Set up Jenkins in a Kubernetes cluster

The goal of this article is to guide you through the steps of setting up Jenkins in a Cluster managed with Kubernetes and hosted in AWS.

Although we provide plenty of reference links about the subject not covered in this article, you will understand more if you already have:

• Basic knowledge about Kubernetes
• Basic knowledge about Docker
• A running EKS Cluster

To add Jenkins to your cluster, follow the steps:

1. Create a Namespace
2. Add ResourceQuota
3. Add RBAC Authorization
4. Create a PersistentVolumeClaim
5. Create the Deployment
6. Create the Service
7. Create the Ingress

Steps 1,2,3 are optional but highly recommended.

Creating a Namespace

First of all, you should create a new namespace in order to separate logically Jenkins from the rest of your cluster.

This is not a mandatory step, but it’s a good rule of thumb, to keep your cluster organized by application.

To create a new namespace called “jenkins” create a file named "namespace.yaml".

apiVersion: v1
kind: Namespace
metadata:
  name: jenkins

Then in the terminal type the following command to apply the changes.

kubectl apply -f namespace.yaml

Adding ResourceQuota

ResourceQuota is used to limit resource consumption per namespace.

Although this is not a mandatory step, it is highly recommended if you want to guarantee that Jenkins does not consume a large chunk (or even all) of your cluster resource, damaging other applications.

To create a ResourceQuota create a file named “resourcequota.yaml”.

kind: ResourceQuota
apiVersion: v1
metadata:
  name: jenkins-quota
  namespace: jenkins
spec:
  hard:
    pods: “5”

Here we are saying that, no matter what, Jenkins namespace should only have 5 pods running at the same time.

Then apply the changes.

kubectl apply -f resourcequota.yaml -n jenkins

Adding RBAC Authorization

Role-based access control or RBAC is a method of regulating access to cluster resources based on the roles of users.

Although this is not a mandatory step, it is highly recommended to limit the permissions of Jenkins will have in your cluster, since it will be able to manage pods.

The first step is to create a Role. In order to create a role add a file named “role.yaml”.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins-role
  namespace: jenkins
rules:
- apiGroups: [“”]
  resources: [“pods”]
  verbs: [“create”,”delete”,”get”,”list”,”patch”,”update”,”watch”]
- apiGroups: [“”]
  resources: [“pods/exec”]
  verbs: [“create”,”delete”,”get”,”list”,”patch”,”update”,”watch”]
- apiGroups: [“”]
  resources: [“pods/log”]
  verbs: [“get”,”list”,”watch”]
- apiGroups: [“”]
  resources: [“secrets”]
  verbs: [“get”]

Now we should create a ServiceAccount, which defines the Jenkins access to the cloud provider, in our case, AWS. We need a specific AWS role for the Jenkins application.

To create a ServiceAccount add a file named “serviceaccount.yaml”.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: jenkins-account
 namespace: jenkins
 annotations:
  eks.amazonaws.com/role-arn: <YOUR_AWS_ROLE>

The last step is to bind the previously created role to the new ServiceAccount. In order to do that we will create a RoleBinding. Here is the file named “rolebinding.yaml”

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-role-binding
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-role
subjects:
- kind: ServiceAccount
  name: jenkins-account
  namespace: jenkins

Then we should apply all the changes.

kubectl apply -f role.yaml serviceaccount.yaml rolebinding.yaml -n jenkins

Creating PersistentVolumeClaims

Jenkins will need some volume to persist its configuration such as users, keys pipelines, etc.

Since volumes in a Kubernetes POD are ephemeral, meaning that in every restart of the POD the files will be lost, we need to create a PersistentVolumeClaim, which requests resources from the cluster to store permanently the data related to the application.

We are going to create a PersistentVolumeClaim called “pvc-jenkins-home” to be used by the master Jenkins pod with ReadWriteMany access, meaning that the volume can be mounted as read-write by many nodes. To do that, add a file named “persistentvolumeclaim.yaml”.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-jenkins-home
  namespace: jenkins
spec:
  accessModes:
  -  ReadWriteMany
 storageClassName: aws-efs
 resources:
    requests:
   storage: 1Mi

Here we are declaring resources that we find interesting for our project, make sure to do the same with yours.

Then to apply the changes run the following command.

kubectl apply -f persistentvolumeclaim.yaml -n jenkins

Creating the Deployment

Finally is time to define the jenkins master pod. To do that we are going to use a Deployment.

We are going to use jenkins/jenkins:lts as our jenkins image. This image documentation says that we need to expose two ports, the 8080, which refers to the Jenkins interface, and the 50000, which refers to Jenkins agents.

We also need to bind the created ServiceAccount and PersistentVolumeClaim to the pod definition and set some probes.

To create the deployment add a new file named “deployment.yaml”.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: jenkins-master
  namespace: jenkins
  labels:
    app: jenkins-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-master
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      serviceAccountName: jenkins
      securityContext:
        fsGroup: 1000
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        - containerPort: 50000
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 300
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 2
          failureThreshold: 5
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
        resources:
          limits:
            cpu: 800m
            memory: 3Gi
          requests:
            cpu: 100m
            memory: 3Gi
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: pvc-jenkins-home

Then to apply the changes run the following command.

kubectl apply -f deployment.yaml -n jenkins

Creating the Services

Now, it is time to expose it as a network service, to do that we are created some Services.

The first service we are going to create is a public one where will expose the port 8080, which is the Jenkins interface. The second service is internal and will expose the port 50000, which will be used when you connect to the Jenkins agent.

To create these services add a file named “services.yaml”.

---
kind: Service
apiVersion: v1
metadata:
  name: jenkins-ui
  namespace: jenkins
spec:
  type: NodePort
  selector:
    app: jenkins-master
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30100
      name: ui
---
kind: Service
apiVersion: v1
metadata:
  name: jenkins-discovery
  namespace: jenkins
spec:
  selector:
    app: jenkins-master
  ports:
    - protocol: TCP
      port: 50000
      targetPort: 50000
      name: agents

Then to apply the changes run the following command.

kubectl apply -f services.yaml -n jenkins

Creating the Ingress

Now that you have your Jenkins up and running with the ports exposed, you need to expose the application to the WEB. In order to do that you should create an Ingress.

In our case, we are also putting a load-balancer between the service and the host (DNS).

All of this is being done by the attributes of annotations bellow. To understand what each annotation is doing, please refer to this link.

To create the ingress add a file named “ingress.yaml”.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ui-ingress
  namespace: jenkins
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: <YOUR_CERTIFICATE>
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/actions.redirect-blueocean: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "Path": "/blue/pipelines", "Query": "#{query}", "StatusCode": "HTTP_301"}}'
spec:
  rules:
  # Here is where you put your domain ex: jenkins.rocks
  - host: <YOUR_HOST>
    http:
      paths:
      - path: /
        backend:
          serviceName: redirect-blueocean
          servicePort: use-annotation
      - path: /*
        backend:
          serviceName: jenkins-ui
          servicePort: ui

Then to apply the changes run the following command.

kubectl apply -f ingress.yaml -n jenkins

Conclusion

After doing all the steps, your Jenkins should be working similar to this diagram below.

In the next article, we intend to teach you how to create pipelines that are triggered by a GitHub pull-request, in other words, how to set up a CI with Jenkins and GitHub.

Here is the repository with all the files you need to put your Jenkins up and running https://github.com/klolivei/jenkins-eks-k8s.

Written by Júlia Arruda and Kaue Silva