Automated Kubernetes deployments with GitLab, Helm and Traefik

Yanick Witschi
Oct 24, 2018 · 15 min read

We all have this one issue: our app is ready, versioned using git where we use different branches for features, bugfixes etc. and it’s all nicely packed up into Docker images and now we would like to deploy these changes to our Kubernetes cluster but it kind of feels boring building the images , pushing them to a Docker registry and then deploying them to the cluster manually. After all that’s what CI/CD is for, right? And also, wouldn’t it be great if we had different environments for different git branches so we could test things easily and isolated and ideally automatically get Let’s Encrypt SSL certificates for every branch URL?

I’ve searched the Internet for a blog entry like this and I found many similar things but none of them actually really fit my needs. Some used some awkward proxy setups using HAProxy which in my opinion needed way too much configuration to run on Kubernetes, others focused too much on a certain app structure and then there are these blog entries where I read five paragraphs and I was already lost.
So, this is my attempt to document this topic and because I know your time is precious and you don’t want to read a whole blog entry only to realize that this was another blog post that didn’t help you with your task, here’s a short list of what I’m going to cover:

  • I’ll cover how to get your code from GitLab (but should work with any CI/CD system) to a Google Kubernetes Engine (GKE) cluster.
  • We’ll have a production system which is our master branch and an arbitrary feature1 branch that represents our feature environment but you’ll be able to have as many feature environments as you like. You’ll be able to access your master branch using your.domain.com and the feature/feature1 branch using feature-feature1.domain.com .
  • We’re using GitLab’s private Docker registry and will thus make sure, GKE can access it.
  • We’re using helm to deploy our application. Your application must thus be prepared to be deployed as a chart! We are not using GitLab’s Auto DevOps feature!
  • We’re using Traefik as our reverse proxy, simply because it’s super easy to use, integrates natively into Kubernetes, has out-of-the-box Let’s Encrypt support, is blazingly fast and because I like French cheese (which is a strong statement if you’re Swiss like me!).

I would like to make one thing clear before I start: If you haven’t used Kubernetes before, never used the CLI tools helm, kubectl or gcloud or you’re kind of new to Docker then I’m sorry but I think you should start with the basics. This will likely be too advanced for you. I’m trying to describe the steps as detailed as I can but I won’t be teaching you fundamental knowledge about all these tools. You have been warned!


Prerequisites

#1: Ensuring GitLab CI/CD can access your GKE cluster

Creating our GKE Service Account

Then we click on Create to get to step two where we can grant this Service Account access to alter our Kubernetes resources. For this, we have to assign the role Kubernetes Engine Developer to the Service Account:

Assign the role “Kubernetes Engine Developer” to your Service Account

Almost done! We only need to generate a key on the last step and download it in the JSON format:

Create a key and download it as JSON file

You’ll end up downloading a JSON file called something like <your-project-name>-<hash>.json. Got this? Excellent!

Now, I have to admit I’m a bit of a paranoid when it comes to encoding issues. Just because more than ten years in IT made me like that. I don’t like to pass things that contain quotes and special characters (like a JSON file could contain) on command line just like that. But we will have to pass this data to our GitLab CI/CD job so we’ll have to place that data into a GitLab CI/CD environment variable. But because I’m paranoid, we’ll base64-encode that data first and will decode it again during the CI job. Whether that makes sense or not to you doesn’t really matter. It’s just my way of trying to stay away from escaping issues with environment variables :-)
Anyway, to encode that JSON key just run $ cat <your-project-name>-<hash>.json | base64 and if you’re a Mac user like me you might just pipe this again so you have the result of it in your clipboard right away like so: $ cat <your-project-name>-<hash>.json | base64 | pbcopy .

We then go to our project on GitLab and navigate to Settings > CI / CD where we can add the encoded key as an environment variable. I give it the name GKE_SERVICE_ACCOUNT but as long as it matches with your .gitlab-ci.yml you’re creating later on, you can name it however you like it best.

This is how it looks like once you stored the environment variable in GitLab

Alright, cool! Now we have an environment variable named GKE_SERVICE_ACCOUNT which contains our base64 encoded JSON Service Account key which we can then use during our CI / CD jobs to access our GKE cluster!

#2: Ensuring GKE can access our private GitLab Docker registry

Creating a personal access token for GKE

Cool, now let’s store this access token as a Kubernetes secret on your GKE cluster. We’ll be using this secret later on. As you can see, the first argument of kubectl create secret is the secret type (docker-registry) and the second one defines the name of the secret. I’m going to name it gitlab-registry so the command you need to execute looks like this, whereas YOUR_PERSONAL_GITLAB_ACCESS_TOKEN_HEREshould be replaced by the token we’ve just generated and I hope you also remember your GitLab username and e-mail address:

$ kubectl create secret docker-registry gitlab-registry \
--docker-server=registry.gitlab.com \
--docker-username=YOUR_GITLAB_USERNAME \
--docker-password=YOUR_PERSONAL_GITLAB_ACCESS_TOKEN_HERE \
--docker-email=YOUR_GITLAB_EMAIL_ADDRESS

Luckily for me, the token only contains unproblematic characters so we don’t need to do any base64 encoding and decoding magic here :-)

#3 Allowing helm (tiller) to manage your GKE cluster

If you haven’t installed tiller yet, you should do so by running $ helm init --service-account tiller as described in the docs.

#4: Assigning a static IP to your GKE cluster

This is how it should look like.

#5: Take a break!

#6: Setting up Traefik

$ helm upgrade \
--install \
--namespace kube-system \
--set rbac.enabled=true \
--set imageTag=1.7 \
--set ssl.enabled=true \
--set ssl.enforced=true \
--set acme.enabled=true \
--set acme.email=<your-email-here> \
--set acme.staging=false \
--set acme.challengeType=tls-alpn-01 \
--set acme.persistence.enabled=false \
--set loadBalancerIP=<your-public-static-ip-from-the-gke-cluster> \
traefik \
stable/traefik

Now there’s a few things to explain here:

  • I’m using helm upgrade with the --install flag instead of using helm install because it is easier for me. It will install it if the release is not there yet and otherwise just upgrade it. This is how I only need to remember one command if I want to later on e.g. add another --set flag but of course, do it the way you prefer!
  • We’re enabling role based access, I recommend you always do that. There’s a reason why there are permissions in Kubernetes :-)
  • imageTag defines the version. I’m using 1.7 which is the latest version at the time of writing this blog post.
  • Replace acme.email with your e-mail address and configure loadBalancerIP so it is set to the static IP we’ve prepared in step #4.
  • You might want to use different settings such as enabling persistence for certificates or enabling the dashboard, debug mode, logging etc. You’ll find more options in the Chart documentation itself. I encourage you to explore more on Traefik, it really is an awesome piece of software!

Finally, let’s create our .gitlab-ci.yml!

#7: Let’s start working!

So what I’ll do is I’ll outline a very basic helm chart which consists of a Kubernetes Deployment, a Service and an Ingress based on Traefik that is everything you need to make your application available to the world. It’s going to be our web deployment. Alongside the basic chart, I’ll post the most simplistic .gitlab-ci.yml you could imagine that works together with this chart. I hope you can then continue from here :-)

So this is how our ./k8s-chart directory looks:

/k8s-chart
/templates
web-deployment.yml
web-ingress.yml
web-service.yml
Chart.yml

Of course, your chart directory might have requirements.yml and requirements.lock , values.yml etc. pp. but again, we stick with the very simplistic approach here!

Okay, so let’s start by looking at the three templates one by one. We’ll start with the web-deployment.yml first:

As you can see, we use two values here web.name and web.image. That’s because we want to have different images (one per branch) and also we want to have different names when we deploy, otherwise how would you know which deployment belongs to what environment, right? Other than that it’s really nothing special here except for spec.imagePullSecrets where you can see that I’ve told Kubernetes to take our gitlab-registry Kubernetes Secret we have configured in step #2. That’s how it’s going to be able to access our GitLab Docker registry. Also, I’ve configured imagePullPolicy to Always so that it really always pulls the latest images. Otherwise if your image tag does not contain e.g. the SHA commit reference but only the branch name, the image tag would stay the same and never get pulled again.

Now on to the web-service.yml which makes our deployment available to the cluster:

This is really super simple: Again, we have to make sure the web.name is used and of course we also have to do that in the selector section so our service finds the correct pods. Other than that, this service just exposes port 80 and that port is named http, that’s it!

Now to the maybe most important one, the web-ingress.yml :

The special thing here is the metadata.annotations.kubernetes.io/ingress.class which is set to traefik. As I’ve mentioned before, Traefik integrates natively with Kubernetes so what you’re doing here is basically telling Kubernetes that Traefik shall handle the ingress, not the native Kubernetes implementation. This is also why the spec might look alien to you because this is now how Traefik can be configured. You can configure loads of stuff but for us, we really just want to tell it to match to a specific host, which we pass again using values (web.host) and forward this to a backend which is our service named web.name and there on the port which is named http. So now I hope you can kind of see how web-ingress.yml goes together with web-service.yml :-)

Nice! What we can do now is by setting the right values for web.name, web.image and web.host we can tell Kubernetes to pull the correct image, deploy it with the correct name and assign the right labels to the Ingress so Traefik will be able to find it based on the host! Half way there!!

So we’re left with our gitlab-ci.yml which needs to do the following:

  1. Build the image and include the branch name somehow.
  2. Push the image to the GitLab Docker Registry
  3. Deploy to the GKE cluster using helm and setting the correct values for web.name, web.image and web.host

Again, I’ll just paste the contents of how it could look like and explain it afterwards:

So starting at the very top, I like to simply use docker:stable as the base image for my CI/CD jobs and then include the Docker-in-Docker Service (docker:dind) because that allows me to basically do anything at every stage as I can re-use docker again and e.g. use $ docker run in one of my steps again. But again, you’re free to choose your own setting.

Then we define two stages, build and deploy which should be pretty self-explanatory.

In variables we can define our environment variables, these are then passed on to the jobs. You do not necessarily have to list them here, you could also just add them to your CI / CD Settings like we did for GKE_SERVICE_ACCOUNT . In fact what happens is that these just operate as fallbacks. If you did not define them in the project settings the job will take what you defined in variables. I kind of like adding everything that actually is configurable here because they serve as sort of documentation for me.

  • IMAGE is likey the most important variable. It’s a combination of two variables that GitLab provides. ${CI_REGISTRY_IMAGE} contains the whole path to your GitLab private registry including the project path, so e.g. registry.gitlab.com/vendor/my-project. This plus /web plus another very useful GitLab environment variable named ${CI_COMMIT_REF_SLUG} will give us our final image name. ${CI_COMMIT_REF_SLUG} is super useful because it contains the git branch name (or tag name) and sanitizes it so that it can be safely used in URL’s, names etc. (checkout more here). So if we push a branch named feature/feature1 we’ll get this: registry.gitlab.com/vendor/my-project/web:feature-feature1, perfect!
  • GKE_SERVICE_ACCOUNT should be clear to you by now :-)
  • GKE_CLUSTER_NAME, GKE_ZONE and GKE_PROJECT could be used to adjust the GKE cluster name, zone and project name if you like via CI / CD settings. But again, you can add defaults here and they’ll be used. They absolutely don’t need to be here. We could also hard-code the values in the init_helm script (we’ll get to that) but I just like to have things configurable :-)
  • URL_REVIEW contains the URL for non-production branches (so e.g. feature-feature1.your.domain.com
  • URL_PRODUCTION contains the production URL

Okay, let’s move on to the CI steps then!

Our build job is not too complicated either. It logs in to our GitLab Docker Registry using the GitLab provided environment variable $CI_BUILD_TOKEN and then builds our image from our ./Dockerfile and pushes it. The important thing here is that we use our environment variable IMAGE which contains our image name and the branch! Given we pushed to our branch named feature/feature1 it will thus build a web:feature-feature1 image and push this to our registry. The docker pull and --cache-from combination is just a CI/CD optimization so it actually pulls a probably existing image first and re-uses the Docker layers it can from there. It’s generally a good idea to do because it’ll speed up your builds dramatically!

So we have the correct image built, tagged with our git branch and pushed to our Docker registry. The only thing left is to deploy it to our GKE cluster. For that, we use two different steps, deployment_review and deployment_production. The main difference being that the GitLab environment is hard-coded to production for our master branch and of course the host is the production URL. GitLab will automatically use deployment_production every time we push to our master branch because we specified only.refs: [master] and for every other branch it will take deploy_review as we specified except.refs: [master].
Possibilities here are endless. You could for example restrict a job to only tags only branches or even a branch regex in case you only want to deploy branches starting with feature/, time to get creative!

Okay, last bit here are the two scripts init helm and then helm upgrade [...].

Our first “problem” is that to be able to deploy the application we need the command line tools helm and because helm requires kubectl we also need that one. And then we also need to have access to our GKE cluster so we also need gcloud to be able to authenticate against our cluster.
If you’re as paranoid as I am, you would go on and build your own Docker image that contains these 3 tools. If you’re not, there are a ton of images out there that contain exactly those :-) For demonstration purposes I’ve used the devth/helm image.

Okay, tools are ready then, now on to the small init_helm script I wrote! It will take our beautifully base64 encoded GKE_SERVICE_ACCOUNT environment variable, decode it and write it to /etc/deploy/sa.json which we use immediately after decoding again to activate the service account using gcloud and get the credentials from the correct cluster, zone and project (which can be configured using environment variables, you remember?). This will configure our kubectl in such a way that it will take our service account for any commands that follow.

To be sure that helm is correctly initialized and runs the latest version, we also run a helm init --service-account tiller --wait --upgrade here. By the way, if you wonder why I’m using --wait for all helm commands: I just want the command to really wait until the tiller pod returns if an action was successful or not. Otherwise our build would not fail if there was an issue with this command which is not desirable during a CI job, is it?

And that’s really most of the magic because what follows is just a regular helm upgrade --install command. It again uses our environment variables to create a deployment named after our git branch and sets the correct values for our helm chart. So given our master andfeature/feature1 branch3w, what you’ll end up with is two helm releases:

  • my-super-app-master
  • my-super-app-feature-feature1

Both running their own deployment, service and ingress and correctly routing their public domains

to your pods/containers with Let’s Encrypt certificates for free.

DONE!

Pretty awesome, if you ask me!
I hope you enjoyed this blog post, let me know if it helped you or if you’re doing things differently so I can continue to learn new things myself :-)

Yanick Witschi

Written by

open-minded Swiss dev, creating stuff mainly with PHP. Especially interested in performance and creating APIs. When not at work, I like sports and Scotch!