How To: Kubernetes Pods as Jenkins Build Agents

Jenkins is an open source automation server used by a lot of people. Part of what makes Jenkins so popular is the robust plugin ecosystem surrounding it, which aim to help solve common problems that plague Jenkins Operators. One of those problems is managing servers that run as Jenkins build agents; managing disk space and patching can become boring and time consuming work. At Vivid Seats, we have solved this problem by replacing static Jenkins agents with Kubernetes pods.

The Setup

To start using the Kubernetes plugin, you will need a running Kubernetes cluster — this guide will show you how to setup a local Kuberentes cluster with Minikube.

To install the plugin, navigate to the Plugin Manager (Manage Jenkins -> Plugin Manager) and click the ‘Available’ tab to search for ‘Kuberentes’. Select the check box next to it, scroll down to the bottom and select ‘Download now and install after restart’.

Once installed, head over to Manage Jenkins -> Configure System and scroll down until you see the ‘Cloud’ heading and update the values to meet your configuration.

Required values

Kubernetes URL: URL of your Kubernetes API Server

• Kubernetes Namespace: Namespace where your build pods will get created
• Jenkins URL: URL for your Jenkins master
• Container Cap: Max number of agent containers that Kubernetes is allowed to run from Jenkins
• Pod Retention: Controls how pods are retained
• Max connections to Kubernetes API : Number of simultaneous connections Jenkins is allowed to send to Kubernetes

Jenkins settings for Kuberntes Plugin

If you want to run this from a Jenkins instance running on your host, you can change Jenkins URL to http://<your_ip_address>:8080 with your machine’s IP address.

Using the Plugin

Create a new Pipeline Job by clicking on ‘New Item’ on the Jenkins home page and selecting the Pipeline type.

On the Configure screen, scroll down underneath the Pipeline settings, and for ‘Definition’ choose ‘Pipeline script from SCM’, which will clone your repo defined into the workspace when your build starts. This is the only setup your job will need as everything else is in your Jenkinsfile.

This plugin integrates very nicely with the Declarative Pipeline syntax, allowing you to define your agents pod spec from your project’s source control, as shown in the gist below.

https://github.com/vivid-lukeloresch/test-pipeline/blob/master/build-pod.yaml

How it works

Here’s the breakdown of what will happen when you click ‘Build Now’ on our pipeline, shown in the Jenkinsfile gist below.

• Jenkins clones your project into the workspace, reads the Jenkinsfile to determine the agent type
• A Kubernetes Pod with the spec defined in build-pod.yaml is created in the namespace defined in the general settings
• Project is build usingmvn clean install on our project which will create a jar file inside the target directory.
• A Docker Image is produced by copying the jar file from thetarget directory

https://github.com/vivid-lukeloresch/test-pipeline/blob/master/Jenkinsfile

The Result

The first part of the build logs show that first the Jenkinsfile is loaded then any configured Jenkins Shared Libraries. Finally, your Kubernetes Pod spec gets sent to the Kubernetes API Server to get created.

Console Output from start of build with internal names removed

Jenkins Pipeline creating a Kubernetes Pod

After all the source code is loaded and the agent is created, your pipeline will begin executing any stages you defined, building our Java app with Maven and packaging it with Docker.

And there you have it, with only a little bit of setup we were able to run our Pipeline in a Kubernetes Pod.

Tips and Tricks

We’ve been using this plugin for a few months at Vivid Seats, and we’ve learned a few key things along the way. Take these tips with you for your setup:

• Set resource requests and limits on each container in your Pod
• Try to use provided images from dockerhub when appropriate, the less you have to maintain, the easier
• Certain optimized images will not have tools like git or curl, so for troubleshooting you can use the jnlp container, which is a container created by default inside your Pod.

Immediate Impact

After adopting this new workflow, we’ve been able to reduce our fleet of traditional Jenkins agents from six large VMs to only two (5 executors on each with 32GB RAM). Maintenance has also become a pain-free experience. No longer do we need to patch and upgrade agents; we can simply change the docker image in the agent’s config. Another benefit we’ve seen is that developers are quickly able to test and create new products without having to spend hours carefully adding new software or updating dependencies on the agent machines.

Was this information helpful?

We hope sharing the details of how we have been using the Kubernetes plugin for Jenkins at Vivid Seats can help others along the way.

Additionally, we are actively hiring for several roles on our engineering team. If you’re interested to learn more about open positions, check out our careers page at vividseats.com/careers for more information.