Kubernetes native Jenkins Operator

self-managed Jenkins on Kubernetes, co-authored with Tomasz Sęk

jenkinsci/kubernetes-operator

What’s the Jenkins Operator?

The Jenkins Operator is a Kubernetes native operator which fully manages Jenkins on Kubernetes. It is easy to install with just a few manifests and allows users to configure and manage Jenkins on Kubernetes.

It provides out of the box:

• Integration with Kubernetes — preconfigured kubernetes-plugin for provisioning dynamic Jenkins Slaves as Pods
• Pipelines as Code — declarative way to version your pipelines in VCS
• Extensibility via Groovy scripts or Configuration as Code plugin-customize your Jenkins, configure OAuth authorization and more
• Security and Hardening — an initial security hardening of Jenkins instance via Groovy scripts to prevent security vulnerabilities

Problem statement and goals

We’ve spent more than two years maintaining many instances of Jenkins on large multi-tenant Kubernetes clusters (hosted mostly on AWS).

The main reason why we decided to write the jenkins-operator is the fact that we’ve faced a lot of problems with standard Jenkins deployment using Helm. Official Jenkins Helm Chart is not production-ready and requires many additional changes to be made. Helm itself is also not ideal.

Instead of endlessly fixing production issues with incompatible plugins, volumes, etc. we prefer to focus on configuration, not the details of manual deployment and lifecycle management — that’s an operator’s job.

Some of the problems we want to solve:

• volumes handling (esp. AWS EBS volume attach/detach issues)
• installing plugins with incompatible versions or security vulnerabilities
• better configuration as code
• lack of end to end tests
• handle graceful shutdown properly
• security and hardening out of the box
• orphaned jobs with no JNLP connection
• not obvious errors, hidden from the end-users
• lack of backup and restore for jobs history

How it works

The jenkins-operator design incorporates the following concepts:

• observe all changes of the manifests and maintain the desired state according to the Jenkins Custom Resource manifest
• implement the main reconciliation loop which consists of two main phases — base and user

Jenkins Operator workflow

Base reconciliation phase is responsible for ensuring base Jenkins configuration, like Jenkins Master pod, plugins, hardening, etc.

User reconciliation phase is responsible for ensuring user-provided configuration, like custom Groovy scripts of Configuration as Code plugin files.

simplified reconciliation loop

For more details please take a look at How it Works section.

Jenkins Operator in Action

This is a step by step walk-through for jenkins-operator deployment.

Requirements

To run jenkins-operator, you will need:

• running Kubernetes cluster and kubectl installed

Configure the Custom Resource Definition

Install Jenkins Custom Resource Definition:

kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkinsio_v1alpha2_jenkins_crd.yaml

Deploy Jenkins Operator

Apply the Service Account and RBAC manifests:

kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/service_account.yaml
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/role.yaml
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/role_binding.yaml
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/operator.yaml

Watch jenkins-operator instance being created:

kubectl get pods -w

Now, jenkins-operator should be up and running in the default namespace.

Deploy Jenkins

Once jenkins-operator is up and running let’s use it to deploy an actual Jenkins instance. Let’s use the example below:

apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
  name: example
spec:
  master:
   image: jenkins/jenkins:lts
  seedJobs:
  - id: jenkins-operator
    targets: "cicd/jobs/*.jenkins"
    description: "Jenkins Operator repository"
    repositoryBranch: master
    repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git

Apply Jenkins Custom Resource:

kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkinsio_v1alpha2_jenkins_cr.yaml

Watch Jenkins instance being created:

kubectl get pods -w

Get Jenkins credentials:

kubectl get secret jenkins-operator-credentials-example -o 'jsonpath={.data.user}' | base64 -d
kubectl get secret jenkins-operator-credentials-example -o 'jsonpath={.data.password}' | base64 -d

Connect to Jenkins:

kubectl port-forward jenkins-operator-example 8080:8080

Got to http://localhost:8080

basic Jenkins UI screenshot

Now, you can prepare your own pipelines and update spec.seedJobs section in your Jenkins manifest.

For more details take a look at Getting Started guide.

Contribution

We’d love to hear your feedback and understand how do you use Jenkins on Kubernetes. Let’s shape Jenkins Operator future together!

Feel free to file issues or pull requests.