Serverless all the things — GitOps using Jenkins X on AWS EKS
This blog post will be focused on the possibilities and challenges which come with setting up and using Jenkins X. I still have to try out a lot to give a final opinion. I will be publishing a second part, where I will present a fully working Jenkins X distribution on AWS.
Probably most of the DevOps community knows how to setup and work with Jenkins. The well known CI/CD tool has established itself in the community over the years due to its versatility and integration options. I personally had to face decisions on which CI/CD tool to use. The number one fallback (or the base of the argument so to say) always has been (and will probably still continue for some time) Jenkins. There will always be the one dependency, the one use case, the one problem you account for, for which other tools are just not open or flexible enough to provide you with long lasting accountability. Of course there are many alternatives and I encourage you to check them out (to get you started: GoCD, CircleCI, TravisCI, GitlabCI).
I’m going to have a look at a extremely viable option for Kubernetes clusters: Jenkins X
Welcome the era of GitOps
I started investigating in Jenkins X to broaden my knowledge about “Serverless” applications and use cases.
Jenkins X is not serverless per-se as it is a combination of many different open-source tools (running as pods) inside a Kubernetes cluster.
Let us start by prefacing this blog entry with my requirements:
- I want to run a cloud native CI/CD tool inside a Kubernetes cluster on AWS (EKS). I want to create the infrastructure using Terraform separately from the JenkinsX CLI, which could create one for you
- I want smooth integration into publicly available SCM’s like Github and also into private instances like a GitLab server on-premise
- Everything must be defined as code (IaC) for idempotent and repeatable deployment
When you start working with Jenkins X, you will encounter the jx CLI (how to install). Every set-up operation against your cluster is run using jx. The recommended way to start is as follows:
You start with going to https://github.com/jenkins-x/jenkins-x-boot-config. These are the source files needed to setup (boot) the Jenkins X into your cluster. You should fork the repository and rename it to the established naming convention (environment-<cluster-name>-<stage> eg.: environment-cicd-dev). Clone the repository and navigate inside. You will start using the first jx command: jx boot (jx boot replaces jx install. It will be deprecated starting Feb 1, 2020).
First lessons learned
The first problem I encountered was the connection from jx to the AWS EKS cluster. This is caused by your local kubectl not being configured against the right cluster (cluster is unreachable). On AWS you can create a valid kubeconfig like shown in this doc.
Moreover, configuring Jenkins X as code can get a little bit tricky. A basic configuration (as found here) is really straight forward. Once you start to fiddle with some of the possibilities, it gets more tricky tho. For example: I never managed to use the vault as the secretStorage. The concept of using and configuring the vault option in AWS is not clear to me solely through the given documentation. Basically its trial and error.
You can have a look at my final jx-requirements file:
Please note: I had to make a few adjustments to properly function with EKS. I had to add the ECR (AWS registry) manually and give my EKS node group the proper permission to access the ECR.
Important: You have to give your AWS EKS node group (or fargate profile) the proper permission for every operation Jenkins X demands. E.g.: For jx to store your logs in an S3 bucket, the node group needs S3 permissions.
Further setup
While jx is booting up, the CLI will prompt you multiple times for some configuration. There is probably a silent way to install Jenkins X, but I haven’t investigated into that as the CLI does not give any more options (as opposed to the deprecated jx install command). You will be prompted to setup a CI/CD bot user which will access your repositories in order to make changes with prow (the GitOps tool).
After that is done (and you crossed your fingers for long enough), all the tooling will be installed inside your cluster. It wasn’t for me at first because my cluster didn’t have enough resources to scale. I increased my worker nodes from 2 to 3 and changed the instance type from t2.small to t3.medium and everything worked like a charm.
You can check for the pods using: kubectl get pods
You can also check for the created kube ingress with: kubectl get ingress
Depending on your configuration you should have a few applications being exposed by Kubernetes.
Problems with external-dns
I encountered some problems connecting the generated ingress from Jenkins X to my Route53 hosted zone using a load balancer (which is also supposed to be generated by jx, subsequently by Kubernetes). After I went through the files for the jxing component (jx ingress), I noticed the use of a network load balancer (NLB) being configured in the chart values.
EKS will recognise the Kubernetes annotations and create AWS native objects (like load balancers used for ingress). After changing the network load balancer to an application load balancer (ALB), jx generated everything nicely. The last thing I had to do is to create an entry in the Route53 hosted zone, which routes all traffic (wildcard CNAME) to you ALB (alias).
For that, take a look at your jx-requirements.yml file and locate your domain entry. You will create a record set inside your hosted zone which looks something like this:
Impressive understanding of EKS
Although jx officially is only tested against the Google Cloud Kubernetes engine (GKE), it worked surprisingly well for AWS aswell.
Like shown above, the templates take in account the creation of AWS native components only using Kubernetes annotations. Thats not only done with jx ingress, but also with other applications like certificates (which I will cover in my next post).
Conclusion
So far I’m impressed of a true GitOps project which definitely deserves the attention. You have quite a lot to configure, which is a curse and a blessing. In my opinion its not ready to be packaged and shipped as a finished product. You will need to know what you are doing, as you hit a wall of configuration rather quick. But the payoff is double the price i think.
But why? Why can’t we just use simple CI/CD tools like the ones established over the years? In my next post I will show you some more configuration and utility options (like certificates) and I will show you the benefits of GitOps with Jenkins X in action.
