Make your very own Kubernetes cluster with Raspberry PIs

Mofizur Rahman
Jan 1, 2019 · 7 min read

This was written over 26 months ago. Which is a lifetime in cloud native years. Many things probably does not work as described here. Please find more up-to-date tutorials if this topic is still of interest. I hope to rebuild this again with the new 2 core 8 Gb ram pis that is available now.

Over the winter break I had some free time to deploy a Kubernetes cluster using raspberry pis. The documentation around this is quite good but I wanted to make a super beginner guide on this.

What you need (must haves)

What you want (Nice to have)

Software Requirement

Pre Setup

7. You should see something similar. Go to Network Options > Hostname. And change the hostname to anything you want. I named mine k8s-master-1, k8s-node-1, k8s-node-2 and k8s-node-3.

8. Once you are done with that, you should set up Localization Options. Although it is optional for our use-case. It is just good practice. Also without it you won’t be able to enable wifi.

9. Once done the pi will reboot.

10. ssh in again. We will give our device a static ip so it keeps the ip between reboots.

cat >> /etc/dhcpcd.conf

Find your device IP. It will be in the form of x.x.x.x. You can find your device ip (on a mac) using ifconfig | grep inet . My mac ip was 192.168.0.26 so my ip format is like 192.168.0.x .

Paste the following code block

interface eth0
static ip_address=x.x.x.y/24
static routers=x.x.x.1
static domain_name_servers=8.8.8.8

Where x.x.x is same as your device ip and y is the ip you want. I put 100 for master and 101,102 and 103 for the rest. Reboot with sudo reboot

When the devices turn on they should reflect the new IPs.

This concludes the initial setup. Next we do the general device setup.

Common Setup

This bit of setup is common for all the raspberry pis. Using sync-pane from tmux is perfect here.

curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker && \
newgrp docker

2. We need to then disable swap. Kubernetes requires swap to be disabled. More info about it on this pull request.

sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove

We can check swap disable was a success by the following command returning empty

sudo swapon --summary

3. Next we edit the /boot/cmdline.txt file. Add the following in the end of the file. This needs to be in the same line as all the other text in the file. Do not create a new file.

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

4. Reboot with sudo reboot

5. SSH in again. Edit the following file

/etc/apt/sources.list.d/kubernetes.list

add the following in the file

deb http://apt.kubernetes.io/ kubernetes-xenial main

Add the key

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

If it works, it will output OK

6. Update with new repo, which will download new repo information.

sudo apt-get update

7. Install kubeadm it will also install kubectl

sudo apt-get install -qy kubeadm

This concludes the common setup.

Master Node Setup

This bit of setup is just for the master node.

sudo kubeadm config images pull -v3

2. We will be using Weave Net as a network overlay.

sudo kubeadm init --token-ttl=0

The - -token-ttl = 0 makes sure our token doesn’t expire. This is not a good practice and should not be done in production.

3. This step takes a long time. (15 mins almost). Once its done it will give a snippet of code to run. Run it to set up kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4. You should also see the join-token. You need this token for other nodes to join your network. It will look something like this

kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea

5. Incase you loose this line or can’t find in the terminal, you can find the token in the following way. kubeadm token list will show you all available token. From there you can find the token. Finding the sha hash is a bit trickier. But you can get it by

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 

This long and convoluted command gets the sha256 for the certificate.

6. Install the Weave Net network driver

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

7. Check Everything worked.

NAMESPACE    NAME                                  READY  STATUS
kube-system coredns-86c58d9df4-9hx5c 1/1 Running
kube-system coredns-86c58d9df4-nfgk5 1/1 Running
kube-system etcd-k8s-master-1 1/1 Running
kube-system kube-apiserver-k8s-master-1 1/1 Running
kube-system kube-controller-manager-k8s-master-1 1/1 Running
kube-system kube-proxy-4k2mc 1/1 Running
kube-system kube-scheduler-k8s-master-1 1/1 Running
kube-system weave-net-5rmmn 2/2 Running

You want the status to be running. If its not, something went wrong. (If its pending or container creating give it some time). But if you see it stuck for too long (3–4 mins) trace your steps back and reconfigure.

8. On the master and all the workers run the following command

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Reasoning for this is explained better in this issue

Worker Node Setup

sudo kubeadm join --token <token> <master-node-ip>:6443 --discovery-token-ca-cert-hash sha256:<sha256>

2. After a few moments, run

kubectl get nodes

and you should see something like

NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1 Ready master 11h v1.13.1
k8s-node-1 Ready worker 11h v1.13.1
k8s-node-2 Ready worker 11h v1.13.1
k8s-node-3 Ready worker 11h v1.13.1

And with this the cluster setup is done.

Additional Steps

We can access our cluster now, which is awesome. But we can only do it while ssh’ed in the master node. We should change that.

We can copy the config from the master to our local machine. Use scp . More info on this on this article.

scp pi@x.x.x.100:.kube/config .

Running the command from you local directory will copy the config file in that directory from master-node.

If you have kubectl installed in you local machine, you can then setup kubeconfig by either overriding the config file in $HOME/.kube/config or you can add the file on top of the existing config (if you have it) by

export KUBECONFIG=<location to config from pi>:$HOME/.kube/config

This way you can keep you kubeconfig and switch to rpi cluster config when needed. You can also look into adding the config in the file instead of overriding it. More info here.

This tutorial is takes inspiration and guidance from Alex Ellis’s work. There are a few other tutorials and blog posts listed on Alex’s repo and are worth checking out for more k8s on rpi goodness.

This is a pretty decent cluster that one can use for learning purposes. With some volume mounted you can do some pretty cool stuff. I will try out some stuff in the upcoming days. Will also try to get the dashboard running.

NYC⚡️DEV

NYC Developer Community