Jenkins X internals — I

10 min readJun 14, 2020

What if there was a tool that could automatically test and deploy your code to the Kubernetes? A tool to create and configure your Kubernetes cluster with multiple environments and fully functional Kubernetes native build pipelines?

A tool that can grab your existing project, add Kubernetes related templates and container setup, and a ready-to-run CI/CD pipeline using best-in-class GitOps principles?

Would it not be great if we can achieve all this — without investing in a complex implementation process, but at the same time have the power to modify all the components to suit our needs?

The good news is there are tools to achieve this goal and Jenkins X is one such tool.

What Jenkins X is?

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Tekton as the underlying pipeline engine.

Jenkins X allows us to harness the power of Kubernetes by simplifying the complex processes involved in Kubernetes and it’s ecosystem into simple concepts that can be quickly adopted. It helps us by guiding throughout our entire software development lifecycle from GitHub to Kubernetes pods.

Jenkins X flavor

Jenkins X comes up with two flavors:
1. Static (Deprecated)
2. Serverless

When Jenkins X first was created, it used to install Jenkins, which now has been replaced with serverless pipelines. This “static” Jenkins installation is now already deprecated, so we can think about only the serverless Jenkins X which is the next generation Jenkins that redefines the process of CD based on GitOps best practices in Kubernetes.

What we get after the installation:

A Kubernetes cluster
A few environments and namespaces
A few git repositories corresponding to the environments
Ingress that creates a load balancer which in turn exposes the services inside the cluster to the external world
A few other things that are essential for making the whole CD process functional

Jenkins X is opinionated. It enforces what it considers to be “best practice” principles on application development in Kubernetes. Everything is based on GitOps, which allows customizing the process as per the need. So we can see at this point that it is not about only CI/CD it is more than that.

Jenkins X environment

After the installation, Jenkins X provides three environments by default.

development
staging
production

The development environment is where the build related things happen. This is where components like Prow, Tekton, etc runs to facilitate the process of build and promotion.

In the staging environment by default the automatic promotion happens. It means once our changes are merged into the master branch then it will be built, tested, and then be promoted to the staging environment so that we can check before finally getting the changes to the production environment. Jenkins X provides us with environments with the promotion policy. The environments that have automatic promotion enabled will get the changes after getting merged into the master.

Kubernetes namespaces

Each Jenkins X environment gets corresponding Kubernetes namespace. We can easily verify that by running jx get env

NAME           LABEL          KIND        PROMOTE NAMESPACE     
dev            Development    Development Never   jx           
staging        Staging        Permanent   Auto    jx-staging    
production     Production     Permanent   Manual  jx-production

We can summarize what we just described in the following picture. This is what Jenkins X will setup for us.

We can see preview environments as well. That one is created once we create a PR against the master branch and allows us to check the changes in an isolated environment kind of giving us the flavor of a real app.

Adding applications to Jenkins X

How does Jenkins X set up the whole process? As a developer, we would just want to push our changes to the repo and that should be reviewed and after getting merged it should be built, tested, and deployed to the staging environment in an automated way. And Jenkins X does help us to set up the whole process like this. To make that happen Jenkins X needs to know about the project and set up a corresponding build pipeline for that which will be triggered with the help of the GitHub webhook.

Jenkins X has two ways to set up the whole process by

Importing an existing project into Jenkins X
Creating a quickstart project using the premade build packs which contains the necessary templates for a particular language and framework.

What we get after importing/creating project with Jenkins X

CD pipeline to Kubernetes cluster
Github webhook
A mechanism to promote a release to different environments
A way to review a pull request

GitOps principles applied to Jenkins X

Jenkins X has been developed based on something called GitOps principles. What is GitOps? In short, it is a way of doing continuous delivery. Everything happens based on changes in git, the defacto source code repository. It means everything should be there in the git repository. Everything means both the application and the environment-related code. Infrastructure and application should be defined with declarative syntax, not with the imperative. Git should be the single source of truth for both the application and infrastructure. If any change is required in the application or infrastructure that should be done through pushing the changes to the corresponding git repo and webhook should trigger the build and deployment.

Now, this GitOps principle is the answer to the question of why there is two separate repo for application and environment.

Viktor Farcic in his book The DevOps 2.6 Toolkit: Jenkins X described 10 commandments of GitOps. These commandments are very important for implementing GitOps based CD process.

Git is the only source of truth.
Everything must be tracked, every action must be reproducible, and everything must be
idempotent.
Communication between processes must be asynchronous.
Processes should run for as long as needed, but not longer.
All binaries must be stored in registries.
Information about all the releases must be stored in environment-specific repositories or
branches.
Everything must follow the same coding practices.
All deployments must be idempotent.
Git webhooks are the only ones allowed to initiate a change that will be applied to the system.
All the tools must be able to speak with each other through APIs.

Now, these commandments should make the above figure much more clear why there are different environments and why the build and promotion are done separately and why release and promotion related information is stored in git.

Due to GitOps principles, we can do everything using git from pushing changes to the promotion of release to the desired environment. All we have to do is create a pull request and approve and merge. That’s it. Build, test deployment will be done automatically by Jenkins X.

How does a simple PR and approval of that PR can trigger build and promotion automatically

Jenkins X does a lot of heavy lifting tasks by creating a bundle of components and making them work as a unit to implement CD in Kubernetes.

To understand the process we have to know a few components from the high level.

Prow
Jenkins X pipeline operator
Tekton

Prow

Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via /foo style commands, and automatic PR merging.

Here in the Jenkins X world Prow is the component that receives the request from the webhook. When we push changes and create PR or merge the PR or write commands in the comment section the corresponding webhook sends the request to prow hook which in turn triggers the next process of the CD pipeline. Prow consists of quite a few components like Deck, Hook, Crier, Tide, etc.

Prow receives the request and acts according to that. It may assign PR, approve and merge PR, run tests, and other git related actions.

When it receives a request for a pull request event then it delegates that request to another component called Jenkins X Pipeline Operator. This component will run the build pipeline and then prow will report the status of the build back to the Git.

Pipeline Operator

Pipeline Operator is the component that simplifies the continuous delivery process by translating the declarative, easy to understand pipeline into Tekton pipeline. Tekton pipelines are very low level and quite painful to write and that’s where the Pipeline Operator comes into the play. It takes our jekins-x.yaml file from the repository and transforms that to Tekton pipeline and tasks and then Tekton does all the heavy lifting tasks for the build process.

Tekton

Tekton is a powerful yet flexible Kubernetes-native open-source framework for creating continuous integration and delivery (CI/CD) systems. It lets you build, test, and deploy across multiple cloud providers or on-premises systems by abstracting away the underlying implementation details.

Tekton is the build engine in the Jenkins X ecosystem. This is the component that runs the build, test, create docker image, helm charts, etc, and push to the respective registries and deploys the release to pull request environment or to the automatic promotion enabled environment like staging environment.

Binary registries

Docker registry
Chart museum
Nexus

While running the pipeline Tekton produces binary data like docker image, helm chart, and shared library. It stores those data in the respective registries. So for docker image, it stores in the docker registry, for helm chart it stores it in chart museum and for shared libraries, it stores in nexus.

Whenever there is a push in the PR branch or master branch, a build is triggered and Tekton creates a pipeline run for each of the builds.

So the flow is something like this

Git → Webhook → Prow → Pipeline Operator → Tekton → Build, test, push to registries → Promotion

If we visualize it would be like below

Application and Environment Repositories and Pipelines

Application repository contains all the info related to the application like source code, build, and deployment-related info in a declarative syntax like YAML, etc.

Environment repo contains info related to the environment. Like which releases are currently running in the environment, versions, etc.

So it means two repositories have two dedicated pipelines. Per application repo, we have one pipeline and per environment, we have another pipeline. So to successfully deploy the release to the desired permanent environment these two pipelines run subsequently to make CD successful.

We can easily verify the pipelines using jx get pipelines

Name                                      URL LAST_BUILD STATUS
git org/app/master
git org/environment-jx-staging/master     N/A N/A        N/A   
git org/environment-jx-production/master  N/A N/A        N/A

We as an engineer pushes changes to the application repository and create a pull request. That’s it. Then as per the above concepts, it will trigger the application build and create a dedicated pull request environment for that PR which means a dedicated namespace. We can access the app in that environment as it provides a dedicated URL to that app. There we can test and verify.

In the GitHub, prow will report back with the status. If it is successful it will show an URL to that PR environment.

Now if we merge then the code is pushed to the master branch which in turn triggers the associated pipeline with that repo and build, test, push binaries to registries and then promote the release.

Now here is the interesting part. The promotion phase creates another pull request against the master branch of the environment repository. Before creating the pull request what it does it changes the requirements.yaml file by inserting/updating the latest version for the corresponding app. Something like this. requirements.yaml is the file that contains the whole definition of the environment.

- name: appname
  repository: http://jenkins-x-chartmuseum:8080
  version: 1.0.86

Now, this pull request in the environment repository starts yet another build pipeline and what it does is it approves the pull request and merges it to the master branch of the environment. This happens automatically.

This merging into the master triggers yet another build pipeline and finally, that pipeline promotes the appropriate version to the staging environment with the help of a command sh “jx promote -b — all-auto …”

Now there is one benefit of mandating infrastructure in a separate repo. Every change to the infrastructure is recorded in git and hence every action is reproducible and reversible. If anything bad happens we can simply create a PR by reverting the current changes and it should trigger the necessary build pipeline. It is the recommended approach of GitOps and Jenkins X implemented that nicely.

So finally we can summarize the whole process of getting our desired changes to the infrastructure by the following steps:

Conclusion

In this article, we took a conceptual point of view on Jenkins X and we saw how it allows our code to travel from our laptop to the Kubernetes cluster.

We also saw how Jenkins X applies GitOps principles, integrating with your version control system to handle building and deploying applications.

If you need a fully functional simplified software development lifecycle based on GitOps principles that simplify the complexity of managing apps in Kubernetes, don’t forget to try out Jenkins X.

References