CI / CD on a Raspberry Pi Kubernetes Cluster

TL;DR

I made my own Kubernetes cluster, hosted on 2 Raspberry PI 3 B+ on which I deployed my website using CircleCI.

Check out the code for project, the pipeline and the site itself.

Not long ago I came across this great article about how one can setup their own, self-hosted Kubernetes cluster on a Raspberry Pi “rack”. I was quite intrigued by the idea and decided to build my own cluster since I already had 2 dusty Raspberry PI 3 B+ lying around.

I got myself a nice acrylic cluster case, a 5-port 60W USB charger and built the physical thing in about 15 minutes.

The Raspberry PI rack (not the best picture, I know).

After having the whole thing ready, I decided to go with Ubuntu Server 19.10 as the base image for both boards because unlike Raspbian, Ubuntu supports the ARMv8 architecture (necessary for the MongoDB image, for example). I’ve set both images in headless mode by adding a file called ssh in the boot card root.

I made sure that I’ve enabled ssh key-based authentication before doing anything else.

$ ssh-copy-id -i <path-to-your>.key <user>@<raspberry-pi-ip>

After this, I’ve added the following entries in the /boot/firmware/nobtcmd.txt file on each node:

cgroup_memory=1 cgroup_enable=memory

You’ll need this later on when you install Kubernetes.

Next, I’ve made sure to disable all the sleep / suspend / hibernate services in Ubuntu so the cluster won’t go offline when it’s not used. You can do that by running the following command on each node, as root:

$ systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

Setting up the actual Kubernetes cluster was a piece of cake thanks to the great k3sup tool. Basically you only need to know the IPs of the used machines, then run the following commands on your machine:

# this is how you setup the master machine
$ k3sup install --ssh-key <your-ssh>.key --ip <ip-for-the-master> --user <login-username>
# this is how you setup the slave node(s)
$ k3sup join --ssh-key <your-ssh>.key --ip <ip-for-the-slave-node> --server-ip <master-node-ip> --user <login-username>

And that’s basically everything you need to do to setup your Kubernetes cluster.

After having everything set up, I decided that as a first project to use this, I would host my personal website (well, a subdomain for now) on this cluster. And because I’m really into fully automated CI / CD pipelines that would handle everything on each git push, I also decided that I would go full out on:

• Docker-izing my project,
• use a private Docker registry to host my images
• setup a CircleCI pipeline that would install the app using Helm on my in-the-office-hosted Raspberry PI Kubernetes cluster

Creating a Docker container for my project

This step was quite easy (or at least that’s what I thought) since my website is just a simple HTML page which uses a few assets (fonts, images, css, js). I’ve used the official nginx docker image initially for this, but unfortunately that’s not compatible with Raspberry PI’s architecture, so after some fiddling around, I came across the arm64v8/nginx build which pretty much solved all the issues that I had before. To make sure I still keep the normal Docker image, I created a separate Docker file called Dockerfile.arm64v8.

# Dockerfile.arm64v8
FROM arm64v8/nginx
COPY assets /usr/share/nginx/html/assets
COPY assets/js/env.arm.js /usr/share/nginx/html/assets/js/env.js
COPY package.json /usr/share/nginx/html
COPY index.html /usr/share/nginx/html/index.html
CMD [“nginx”, “-g”, “daemon off;”]

You can build Docker images based on a file different than Dockerfile by using the -f flag in the docker build command:

docker build -f <some-other-dockerfile> -t <tag> .

Using a private Docker registry

This is quite an important step since all Helm deploys will use pre-built images pulled from a remote repository. Fortunately, there are quite a few options to do this if you’re not into hosting a registry by yourself. I went for TreeScale because it offers unlimited repositories for free. The only thing that I don’t like about TreeScale is that the UI doesn’t seem to present up-to-date data.

To use TreeScale, you only have to sign up on the site, then login into your private docker registry:

$ docker login repo.treescale.com -u <user> -p <password>

After that, every image that you tag with treescale.com/<username>/<repo>:<version> will be uploaded to the repository on each docker push <image> run.

Setting up a CircleCI pipeline

I chose to use CircleCI for this project because I already had it hosted on Github (with which CircleCI integrates really well) and I really like the idea behind CircleCI’s Orbs.

The pipeline for this project had to be able to do the following:

• build both the normal Docker image and the ARM one
• authenticate and push both images to the private Docker registry
• connect to the Kubernetes cluster
• upgrade / install the app

Fortunately, all these steps are really easy to set up in a .circleci/config.yml file, especially if you use Orbs for configuring kubectl and helm.

To handle all the sensitive data such as docker registry credentials and kubernetes credentials, I’ve used CircleCI’s environment variables.

The CircleCI environment variables used for the build-deploy pipeline.

When it comes to deploying applications on a Kubernetes cluster, I’m always going for Helm charts. It’s a really nice way of keeping your infrastructure as code bits tidy and organized. For this project, I went for a basic setup, by mostly using the default chart that Helm generates with the helm create chart command. The cool thing about using Helm on your K8s apps is that you can use the --set flag to set custom values on each run and that’s exactly what I’ve used to make sure that I install the latest version of the app (using the $CIRCLE_BUILD_NUM environment variable).

helm upgrade homepage ./chart --set image.version=armv$CIRCLE_BUILD_NUM --install

Having this in place, the only thing that was left was to run the pipeline and get the site up and running in the cluster.

Conclusion

Implementing your own CI / CD setup on a K8s cluster set up on a bunch of Raspberry Pis is a really fun project that I recommend to any DevOps aficionado.

The repository for this project is hosted on GitHub; you can also check out the CircleCI pipeline, but also the site itself.

Cheers!