In my previous article, I gave you the lowdown of what is in my cluster. Now, let's take a look at what it took to get it in place.
I am running a cluster of 3 RPis — 1 Pi4 and 2 Pi 3B+s. Each of those is connected individually to a power supply and a switch. I don’t have any fancy PoE hats, although that would be a very good improvement for my setup. One key thing to mention at this point is storage — we are going to need centralized storage somewhere in the cluster, and I opted for an external HDD, but any mountable storage will do the trick. At this point, the network diagram is fairly simple.
As you can see from the topology diagram, I have a single master node and 2 additional agents. This does introduce a single point of failure into the cluster — if the master node dies, then nothing will be managing the cluster. I have accepted this risk, given I have a single point of failure with my HDD too. I have mitigated it by ensuring both failure points are on a single node, meaning both of the other nodes are nonessential and can be recovered from.
Setting up the Master Node
Before installing the cluster, you need to ensure that any storage attached to the node is attached on boot. How to do that is out of scope for this article, but there are many good resources around to suit any use case. Additionally, all hostnames should be set up across the cluster before installing K3s, otherwise, you can get in a bit of a pickle. Installing K3s is one simple command
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='server --no-deploy traefik --disable servicelb' INSTALL_K3S_VERSION="v1.18.9+k3s1" sh
This snippet will install and start K3s, without Traefik and the built-in load balancer. This allows me to manually install the version of Traefik I want with more control, and to install MetalLb later. I have also pinned the version so I can install the same version across the cluster. I did originally have a later version, but that was giving issues with CPU usage skyrocketing, so thanks to my cluster monitoring I could pinpoint that. At the time of writing, 1.18.9+k3s1 was the latest version I have proven stability for on my setup.
Checking the cluster is up and running is really straight forward. This command will tell you what is up and connected
sudo kubectl get nodes
It should give you something like this if it has worked
Setting up your agents
The first thing you need is to get your node token from the master. Run the following command to get that.
Once you have that, on each of your nodes run the following
curl -sfL https://get.k3s.io | K3S_URL=https://<<MASTER-NODE-IP>>:6443 K3S_TOKEN=<<NODE-TOKEN>> INSTALL_K3S_VERSION="v1.18.9+k3s1" sh -
Once you have done that on each node, check the nodes again, and the output should have changed to reflect this
You now have a 3 node cluster! It is however of limited external use until you install an ingress controller such as Traefik.
Access outside the cluster
The simplest way is to take a copy of your kubeconfig file (found at /etc/rancher/k3s/k3s.yaml) and use kubectl on another machine using that as the config file. Be aware you will need to update the server to not be 127.0.0.1, but the IP address of your cluster master node.
One of the most important things we need to get set up is our ingress controller. Thankfully, that's really simple thanks to a tool called Helm. We will use Helm to install Traefik into our cluster. You can specify a lot of things in the values, but the main ones to look at are the TLS setup for our HTTPS certificates and the loadbalancer config. Each of those will be specific to your setup. For mine, I have set a specific load balancer external IP so that I know I will always get the same one as an entry point, rather than relying on auto-assignment. See here for a base set of defaults. You can take a copy of that and update it for your scenario. When you have that, run the below commands
sudo kubectl create namespace traefik
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik --namespace=traefik --values=traefik.values.yaml
That should now setup Traefik for you. To check, run the following to see the status of the pods, and just look for a running pod with 1/1 ready.
sudo kubectl get pods -n traefik
Finally, to allow access we need a working load balancer. You should have a load balancer service created, but because we have nothing to service it yet, it will be stuck at pending
sudo kubectl get svc -n traefik
This is also fairly simple to install to your cluster. There is a premade YAML file that you can apply straight to your cluster which does most of the work.
sudo kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
The only thing missing is you will need to add a configuration specific to your cluster for the load balancing. I created mine in metallb.yaml:
- name: address-pool-1
And then applied it in the standard way
sudo kubectl apply -f ./metallb.yaml
This gives me a load balancer setup that will dish out IPs in the range 192.168.1.200–192.168.1.254. I also excluded this range from my DHCP pool to ensure I didn't end up with network clashes. Finally, if you look at the running services for Traefik again, you should see that your load balancer now has an external IP that you can use outside the cluster to access it.
Testing it all together
A simple way to test this would be to add an ingress route into the Traefik dashboard. If you don’t have any DNS entries you can point at your cluster, you can make do with hostfile entries for now. This should not be left on for a production-grade cluster because it gives out a lot of information, and is not secured (by default). The following traefik-dashboard.yaml will setup the route
- match: Host(`traefik.internal`)
- name: api@internal
- name: traefik-sso@kubernetescrd
And again apply it the usual way
sudo kubectl apply -f ./traefik-dashboard.yaml
You should then be able to access the Traefik dashboard at https://traefik.internal. It should look something like this
Congratulations! You now have a working Kubernetes cluster ready to service both internal and external requests! Next time we will look at persistent storage options open to us, so we will be able to deploy apps with storage requirements.