Scaling and optimizing Jenkins on Oracle Container Engine
Since the first release of Jenkins in 2011, a lot has happened in the Software Engineering landscape. With the introduction of Kubernetes in 2015, distributed and shared-nothing architectures have gradually become the norm, enabling developers to write resilient and fault-tolerant applications.
Many CI/CD tools have also adopted these architectural styles, and as a result many recent DevOps frameworks coming into the market are heavily based on Kubernetes (Jenkins-X, Tekton, Spinnaker, ArgoCD..).
Despite all these new tools, Jenkins still remains the go-to choice for many Enterprise companies due to its simplicity and community-wide.
In this article I will explain how Jenkins can be scaled and run on Kubernetes, focusing also on some plugins that will help managing the overall solution.
Lastly, there will be a chapter describing how it is possible to optimize resources by deploying Jenkins on Oracle Container Engine, the managed Kubernetes service on Oracle Cloud Infrastructure.
A day in the life of a DevOps Engineer
DevOps Engineers are often considered as some sort of wizards, as they have to understand the whole Software Development Lifecycle process and automate what is worth automating.
Using Jenkins with the usual Controller/Agent architecture might seem the easiest option, however, some issues may arise with time.
While a DevOps project might start off from a small group of developers, as more and more teams decide to adopt DevOps practices the situation becomes more complex.
Jenkins issues at scale
Jenkins is a very powerful tool, although using it at scale could be challenging, I have identified four main problematic areas:
- Scaling: a system is scalable if it is capable of handling the dynamic variability of an environment by growing or shrinking, where the environment here consists of people (development teams) and Jenkins agent nodes.
- Management: everything that is related to the management of a system, including updates of plugins and Jenkins upgrades.
- Standardize: keep configurations and pipelines in order and define build standards in order to reuse as much as possible.
- Resource Optimization: optimization of Jenkins resources to use less resources to run Jenkins at scale.
Running Jenkins on Kubernetes
As more and more teams join a DevOps project, the DevOps engineer needs to instantiate multiple Jenkins controllers and agents instances. This is because not all the teams will accept to share a Jenkins instance, and some applications need a dedicated build server.
Furthermore, as teams can shrink or grow it is difficult to estimate the number of resources needed for Jenkins.
What if it was possible to containerize Jenkins and run it on Kubernetes?
Jenkins Kubernetes plugin makes possible to run Jenkins agents as pods.
Every time there is a new pipeline job, the Jenkins controller will generate a Kubernetes manifest to run the agent on a Pod. This Pod will always have a container named Jnlp in charge of maintaining and securing the connection with the controller.
The manifest can be defined directly on the Jenkinsfile, that is the file where the pipeline is specified, thus giving the DevOps engineer the ability to define as code all the containers and build tools needed for the pipeline.
From a scaling perspective, as Pods are created on-demand, if a development team shrinks/grows there will be only less/more Pods in Kubernetes, which are easy to handle and scale.
In addition, the build tools are not installed directly on the agent nodes, but they are defined as containers in the agent Pod. This gives a tremendous boost in the overall management of agent build tools.
To better understand the concept, let’s consider this scenario:
Two development teams, A and B, are developing some Java microservices and they need some Jenkins pipeline to build a jar. Team A is using Java 8 while team B is using Java 17.
With the Kubernetes plugin, all there is to do is changing the tag of the container providing the JDK for the agent pod in the Jenkinsfile of both team A and of team B.
Jenkins CasC: Configuration as Code
The JCasC plugin is an essential one for provisioning new Jenkins controllers and define their configuration through code (yaml).
To make the advantages of this plugin clear, let’s review this scenario:
A DevOps Engineer needs to provision a new Jenkins controller for development team B. Team A is already using Jenkins and team B needs a similar setup.
Without CasC, the DevOps Engineer would need to provision another Jenkins controller and then configure the instance with the same configurations of team A.
By having all configurations as code, it is possible to standardize configurations across multiple Jenkins controllers, and managing code is far easier and time efficient than configuring manually all the Jenkins controllers.
Jenkins Job DSL: Seed Pipeline
The Job DSL plugin enables a DevOps Engineer to define a seed pipeline, that is, a pipeline capable of generating other pipelines.
As usual, let’s examine a practical scenario:
A DevOps Engineer needs to provision a new Jenkins controller for team A. Everything is correctly provisioned and configured through code, but team A still needs help defining the pipelines in Jenkins.
By defining a pipeline capable of generating other pipelines, the DevOps Engineer will only need to define one pipeline and tell team A to run it to generate their pipelines.
As the seed pipeline is basically code, with Job DSL it is then possible to manage pipelines as code and standardize pipeline definitions through the organization.
As a best practice, it is possible to define pipelines using the Jenkins DSL (Domain Specific Language) directly in the JCasC plugin.
By specifying the creation of a seed pipeline in the configurations, when a new Jenkins controller is provisioned it will also come with the seed pipeline already defined and ready to run.
Remote Jenkinsfile Provider
The last plugin examined here might seem trivial, but it’s something very useful for every DevOps Engineer.
Let’s consider the following scenario:
A DevOps Engineer needs to write a Jenkins pipeline to build Java microservices. This pipeline will be the same for every development team in the organization, as all the code needs to undergo some quality control and security tests before being built. If the pipeline is updated, then all the developers need to have this new version of the pipeline as soon as it is released.
In Jenkins, the Jenkinsfile represents the code where the pipeline is defined and it is by default stored in the Git repository of the source code to build.
For small teams where the developers are also in charge of the build pipelines, this might work fine. But when there is a need to standardize the build process across the organization and relieve the developers of the burden of managing pipelines, then this approach is no more feasible.
In a context where there are 100 Java developers using the same Jenkinsfile and the DevOps Engineer release a new version of the pipeline, the DevOps Engineer would need to notify all the developers and give them the new Jenkinsfile so that they can update it in the source code.
With the Remote Jenkinsfile plugin, it is possible to avoid all this and store the Jenkinsfile in a remote Git repository, away from the source code.
By doing so, the same Jenkinsfile can be used by many pipelines and teams, and in case of a new release then the DevOps Engineer will only need to push the new Jenkinsfile on the remote Git repository.
Rollback of a pipeline release is also easier, as the Jenkinsfile is stored in Git. Furthermore, if only the DevOps Engineers can access the pipeline repository, developers won’t be able to modify the build process, thus securing and standardizing the whole software development lifecycle.
Resource Optimization: Jenkins on OKE
Oracle Container Engine for Kubernetes (OKE) is the managed Kubernetes offering provided by Oracle Cloud. Thanks to the deployment flexibility and price performance offered by OKE, it represents an ideal service where to deploy Jenkins.
Solution Architecture
The picture above shows a high level architecture of a scalable Jenkins deployment on OKE. Each Jenkins controller pod will have a persistent volume to keep pipeline history and plugins installed by the user.
The agent pods in the picture are created on-demand using the Jenkins Kubernetes plugin.
Jenkins on arm instances
As stated in this blog, starting from 2021 Oracle has a partnership for bringing Jenkins on arm processors. Oracle Cloud offers the Ampere A1 arm-based instance that, compared to x86, are more performant and 25% cheaper.
It is possible to create a node pool with Ampere A1 instances in OKE, which represents an ideal place where to place Jenkins controller pods, as they will need to be always on.
Using preemptible instances for Jenkins Agents
A preemptible instance in Oracle Cloud is a Virtual Machine which can be stopped at any moment. The advantage of using a preemptible instance is that there is a reduction in price of 50% compared to normal capacity.
Preemptible instances are ideals for short-lived and fault tolerant applications.
It is possible to create a node pool with preemptible instances in OKE. Furthermore, if the capacity of a node is reclaimed, OKE will automatically replace that node with another preemptible instance.
As on-demand Jenkins agent are short-lived, choosing to run them on preemptible instances is ideal for saving costs.
Deploying Jenkins: Helm Chart
The Jenkins community provides a useful Helm Chart to quickly install Jenkins on any Kubernetes cluster.
By configuring the Chart values, it is possible to include custom JCaS configurations that will be saved as ConfigMap.
The Helm Chart include the following components:
- Controller plugin init container: an init container that will install all the plugins configured in the Helm values by the user. Although it may sound a handy feature, it is recommended to create a custom Jenkins image with all the initial plugins required and disable the init container.
- Config reloader sidecar: a sidecar container that will automatically reload the Jenkins controller in case the JCaS ConfigMap changes. Enabled by default, but can be disabled in case it’s not needed.
- Backup CronJob: a Kubernetes CronJob to perform the backup of the Jenkins controller on an S3 bucket. It is disabled by default.
To run the Jenkins Controller only on arm nodes in OKE, be sure to modify this property in the Helm value file:
controller:
nodeSelector: {"kubernetes.io/arch":"arm64"}As for the Agent on-demand pods, they can be defined in the Jenkinsfile and all pods running on preemptible instance in OKE needs the following toleration:
spec:
tolerations:
- key: "oci.oraclecloud.com/oke-is-preemptible"
operator: "Equal"
value: ""
effect: "NoSchedule"Additionally, configure the nodeSelector for the agents in the Helm Chart:
agent:
nodeSelector: {"oci.oraclecloud.com/oke-is-preemptible":"true"}There are many other Helm values worth configuring in the Helm Chart, and reading the default values.yaml is highly suggested.
Additional suggestions to manage Jenkins at scale
As everything in Jenkins can be defined as code through the use of plugins, it would make sense to also use GitOps practices to manage Jenkins deployments with the Helm Chart. Tools as ArgoCD and FluxCD can help deploying Jenkins for many different teams.
Once everything is in place, it should take the DevOps Engineer around 10 minutes to deploy a new Jenkins instance for a new team.
Conclusions
Although this article explains how to solve many issues when scaling Jenkins and how to optimize costs and performance using a public cloud provider, some aspects of Jenkins still remain unsolved.
From the management perspective, plugins and version upgrades might break the whole Jenkins installation or some pipelines. The only solution for this is testing before releasing a new Jenkins version.
As the DevOps Engineer can now provision a new Jenkins instance on the fly through code, it is relatively easy to test new versions before rolling them out.
In terms of resource optimization, the preemptible nodes on the OKE cluster are still running even if there are no Jenkins jobs.
In case Jenkins is not needed, the nodes on OKE can be scaled down.
For example, if the development team only works during daylight, the nodes can be scaled to zero at night and then restored the following day.
As technology and cloud services progress, more optimal solutions might become available to deploy Jenkins and other Cloud Native workloads.
Nevertheless, a good engineered system is one which is scalable and sustainable for the foreseeable future.
Although Jenkins might look very easy to start with, it’s not sustainable if many teams use it and scaling it requires a complete re-architecture of the solution. Thus, this article might be useful both for seasoned Jenkins administrators and new people approaching the DevOps world so that they can understand the challenges underneath the apparently simple Jenkins architectures.
Happy coding to everyone!
