Homelab Kubernetes Cluster with GitOps

4 min readMar 9, 2022

I recently setup a homelab Kubernetes cluster as a personal learning experience and to function as an additional set of resources for local development. I wrote this up to both serve as a personal reference for later and for anyone else considering a similar path.

This will cover my process from 0 to fully running K3s cluster with GitOps enabled through FluxCD. Short and simple — I won’t go much into the nitty gritty of each component and decision, just enough to get an idea on my thought process.

The goal with the hardware selection was to maintain simplicity and reduce any potential headaches down the road. When first exploring my options for a bare-metal Kubernetes cluster, I looked at utilizing a set of Raspberry Pi devices, however chose not to for 2 main reasons:

ARM64 architecture During my research, I found some different discussions (on Reddit, Medium, etc.) around ARM64 arch and docker images. The TLDR: some open source containers aren’t released for this architecture. You can rebuild the images as needed, but I didn’t want to waste any time on this down the road.
Storage The Raspberry Pi utilizes a microSD for boot, and then with a k3s cluster, it will also need additional storage for its normal operation. I wanted to avoid managing multiple disks and the extra possible clutter.

With that in mind, I looked for an x86 mini computer with M.2 SSD and included RAM (8GB+ preferably). I planned to setup 4 units (1 master with 3 workers), so maximizing compute power for cost was also a factor.

Here’s what I ended up going with:

Hardware List

Awow Mini PC

Includes Intel x86 CPU, 8GB RAM, & 128GB M.2 SSD (that can be expanded)

TP-Link 8 Port Switch
1 ft. Cat 7 Cables (5 pack)
25 ft. Cat 7 Cable
Desktop Stand (to organize the mini PCs)

Setup

OS Installation
K3s
FluxCD

OS Installation

The mini PCs came installed with Windows, so I needed to wipe them and install Ubuntu. We’ll need a USB along with a keyboard and mouse (to attach to the mini PCs) to complete this.

Here are the basic steps for this process:

On separate computer, download appropriate linux iso (I used Ubuntu 20.04 LTS server)
Use Balena Etcher to flash iso to USB
On a mini PC, insert USB and select boot from USB
Follow ubuntu installation prompts
Repeat steps 3 & 4 for each mini PC.

Here’s the full video I followed:

Now, we have our mini PCs installed with Ubuntu, we can move on to configuring the Kubernetes cluster, but first we’ll need to setup the hardware.

Plug in the switch to power and connect the switch to the router using the long ethernet cable
Plug each mini PC to the switch using the short ethernet cables
Log into your router and set a static IP for each server for convenience and later ease of use.

K3s

For my homelab, I chose to use K3s as my Kubernetes distro to keep the system more lightweight due to my hardware constraints.

For installing and configuring the servers, I forked an existing repo that has a set of ansible roles to configure all the servers with K3s appropriately.

The original repo was built for Raspberry Pi, so I removed / change a few of RPI specifics items to work with my x86 servers.

Github Repo: https://github.com/johnjednorski/ansible-k3sup

To run the playbook, you’ll need to complete a few setup steps:

Ansible Playbook Setup Steps

Ensure Ansible is installed

$ ansible --version
ansible [core 2.11.0]
...

2. Setup encrypted secret for playbook

Create vaultfile.yml with cluster_secret and save password in .vaultpw. This is needed for the playbook (and these files are already in .gitignore).

$ cd ansible-k3sup $ mkdir vars $ echo 'cluster_secret: $YOUR_PASSWORD' > ./vars/vaultfile.yml $ echo $YOUR_PASSWORD > .vaultpw $ ansible-vault encrypt ./vars/vaultfile.yml --vault-password-file .vaultpw

3. Configure hosts file in: inventory/hosts.ini

$ cd ansible-k3sup $ mkdir inventory 
# Copy example hosts file from readme to hosts.ini $ vim ./inventory/hosts.ini 
# Configure ansible_host, ansible_user, & k3s_version appropriately

4. Run playbook to install k3s on servers.

$ cd ansible-k3sup $ ./play.sh # Optionally, specify ansible verbose flags to debug issues 
$ ./play.sh -vvv

5. Verify setup succeeded and nodes after configured properly

$ k get nodes --sort-by='{.metadata.name}' 
NAME STATUS ROLES AGE VERSION 
k3s-master Ready control-plane,master 1d v1.21.0+k3s1 
k3s-worker1 Ready <none> 1d v1.21.0+k3s1 
k3s-worker2 Ready <none> 1d v1.21.0+k3s1 
k3s-worker3 Ready <none> 1d v1.21.0+k3s1

FluxCD

Now that we have the K3s cluster setup, we can setup FluxCD. I will be using this environment to become more familiar with Continuous Deployment through GitOps.

These steps will bootstrap FluxCD for my development cluster and create a repo for the initial configuration for this environment. To continue working with this repo, I have modeled my workflow off of this example from Flux which provides a multi-environment setup (i.e. development -> staging -> production)

This bootstrapping step will configure Flux only for the development environment. This step can be repeated per additional cluster / environment.

FluxCD Steps

Ensure Flux client is installed

$ flux --versionflux version 0.13.4

2. Create Gitlab Access token here.

3. Export Gitlab Token

$ export GITLAB_TOKEN=<your-token>

4. Run Flux Bootstrap command

$ flux bootstrap gitlab --owner=$GITLAB_GROUP --repository=$REPO_NAME --path=clusters/development

5. Ensure flux is configured and working properly.

$ flux check 
► checking prerequisites 
... 
✔ all checks passed

Next Steps

So to recap, we:

Provisioned hardware
Installed Ubuntu
Installed & configured K3s Cluster
Installed & configured FluxCD for GitOps

For next steps, I will be setting up configuring various infrastructure components (Kafka & TimescaleDB), CI for my applications, and Flux Image Automation so my build artifacts will be automatically deployed.

Let me know if you have any questions. I wanted to keep this brief and as actionable as possible.