Make your very own Kubernetes cluster with Raspberry PIs

Published in

NYC⚡️DEV

7 min readJan 1, 2019

This was written over 26 months ago. Which is a lifetime in cloud native years. Many things probably does not work as described here. Please find more up-to-date tutorials if this topic is still of interest. I hope to rebuild this again with the new 2 core 8 Gb ram pis that is available now.

Over the winter break I had some free time to deploy a Kubernetes cluster using raspberry pis. The documentation around this is quite good but I wanted to make a super beginner guide on this.

What you need (must haves)

Raspberry PI (2/3). Minimum of 2 (1 master and 1 worker)
Power Cable (Micro USB, same as the old android chargers)
Ethernet Cable (Can be done with wireless, but ethernet cable makes it easier)
Micro SD Card (Minimum of 8gb)

What you want (Nice to have)

Raspberry PI 3 Model b+. 4 (1 master and 3 worker)
Gigabit Ethernet Switch.
Short (1 ft) ethernet cable
Short (1 ft) Power cable
Multiport USB Charging HUB. (I used a 6 port Anker hub)
Cluster Case (Keeps everything nice and clean)
Micro SD Card (32gb each)

Software Requirement

Etcher
Raspbian Stretch Lite
Tmux (Useful for sending command to multiple terminal windows)
Access to the Router Admin or IPScanner software.

Pre Setup

I am following this tutorial to setup. If anything I do is confusing or too slow for your pace feel free to jump there. My goal is to make this version beginner-friendly.
Using etcher (or something similar) burn the os image onto the MicroSD card. You can follow this guide for instruction.
Put an empty file named ‘ssh’ (the file has no file extension or any content in) inside the SD card. This will allow us to ssh into the device out of the box. That way we skip the need for connecting a monitor and other input peripherals.
Insert the SD cards in the raspberry pi and power on the pis. If you check in your router connected device you should see a few new devices pop up. The hostname on all of these would be raspberry. If you can’t see the devices on your router you can use a IPScanner, I used Angry IP Scanner.
Since there are multiple devices that need some setting up, Tmux will come in super handy. Although this can totally be done without tmux. But I had 4 pis to setup and doing the same thing 4 times is not my thing. If you never used tmux you are in for a treat (or torture, or maybe both). This guide will get you started. I will use the synchronize-pane feature and turn it on and off as I need.
At this point, I am hoping you are ssh’ed in all the pis. Type in raspi-config. It is a built in raspberry pi tool to configure the device.

7. You should see something similar. Go to Network Options > Hostname. And change the hostname to anything you want. I named mine k8s-master-1, k8s-node-1, k8s-node-2 and k8s-node-3.

8. Once you are done with that, you should set up Localization Options. Although it is optional for our use-case. It is just good practice. Also without it you won’t be able to enable wifi.

9. Once done the pi will reboot.

10. ssh in again. We will give our device a static ip so it keeps the ip between reboots.

cat >> /etc/dhcpcd.conf

Find your device IP. It will be in the form of x.x.x.x. You can find your device ip (on a mac) using ifconfig | grep inet . My mac ip was 192.168.0.26 so my ip format is like 192.168.0.x .

Paste the following code block

interface eth0
static ip_address=x.x.x.y/24
static routers=x.x.x.1
static domain_name_servers=8.8.8.8

Where x.x.x is same as your device ip and y is the ip you want. I put 100 for master and 101,102 and 103 for the rest. Reboot with sudo reboot

When the devices turn on they should reflect the new IPs.

This concludes the initial setup. Next we do the general device setup.

Common Setup

This bit of setup is common for all the raspberry pis. Using sync-pane from tmux is perfect here.

The main choices for a container environment are Docker and cri-o. We will user Docker, as cri-o requires a fair amount of extra work to enable for Kubernetes. As cri-o is open source the community seems to be heading towards its use The following command installs docker and sets the right permission.

curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker && \
newgrp docker

2. We need to then disable swap. Kubernetes requires swap to be disabled. More info about it on this pull request.

sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove

We can check swap disable was a success by the following command returning empty

sudo swapon --summary

3. Next we edit the /boot/cmdline.txt file. Add the following in the end of the file. This needs to be in the same line as all the other text in the file. Do not create a new file.

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

4. Reboot with sudo reboot

5. SSH in again. Edit the following file

/etc/apt/sources.list.d/kubernetes.list

add the following in the file

deb http://apt.kubernetes.io/ kubernetes-xenial main

Add the key

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

If it works, it will output OK

6. Update with new repo, which will download new repo information.

sudo apt-get update

7. Install kubeadm it will also install kubectl

sudo apt-get install -qy kubeadm

This concludes the common setup.

Master Node Setup

This bit of setup is just for the master node.

Pre-pull images

sudo kubeadm config images pull -v3

2. We will be using Weave Net as a network overlay.

sudo kubeadm init --token-ttl=0

The - -token-ttl = 0 makes sure our token doesn’t expire. This is not a good practice and should not be done in production.

3. This step takes a long time. (15 mins almost). Once its done it will give a snippet of code to run. Run it to set up kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4. You should also see the join-token. You need this token for other nodes to join your network. It will look something like this

kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea

5. Incase you loose this line or can’t find in the terminal, you can find the token in the following way. kubeadm token list will show you all available token. From there you can find the token. Finding the sha hash is a bit trickier. But you can get it by

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

This long and convoluted command gets the sha256 for the certificate.

6. Install the Weave Net network driver

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

7. Check Everything worked.

NAMESPACE    NAME                                  READY  STATUS
kube-system  coredns-86c58d9df4-9hx5c              1/1    Running
kube-system  coredns-86c58d9df4-nfgk5              1/1    Running
kube-system  etcd-k8s-master-1                     1/1    Running
kube-system  kube-apiserver-k8s-master-1           1/1    Running
kube-system  kube-controller-manager-k8s-master-1  1/1    Running
kube-system  kube-proxy-4k2mc                      1/1    Running
kube-system  kube-scheduler-k8s-master-1           1/1    Running
kube-system  weave-net-5rmmn                       2/2    Running

You want the status to be running. If its not, something went wrong. (If its pending or container creating give it some time). But if you see it stuck for too long (3–4 mins) trace your steps back and reconfigure.

8. On the master and all the workers run the following command

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Reasoning for this is explained better in this issue

Worker Node Setup

From each worker node run

sudo kubeadm join --token <token> <master-node-ip>:6443 --discovery-token-ca-cert-hash sha256:<sha256>

2. After a few moments, run

kubectl get nodes

and you should see something like

NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    11h       v1.13.1
k8s-node-1     Ready     worker    11h       v1.13.1
k8s-node-2     Ready     worker    11h       v1.13.1
k8s-node-3     Ready     worker    11h       v1.13.1

And with this the cluster setup is done.

Additional Steps

We can access our cluster now, which is awesome. But we can only do it while ssh’ed in the master node. We should change that.

We can copy the config from the master to our local machine. Use scp . More info on this on this article.

scp pi@x.x.x.100:.kube/config .

Running the command from you local directory will copy the config file in that directory from master-node.

If you have kubectl installed in you local machine, you can then setup kubeconfig by either overriding the config file in $HOME/.kube/config or you can add the file on top of the existing config (if you have it) by

export KUBECONFIG=<location to config from pi>:$HOME/.kube/config

This way you can keep you kubeconfig and switch to rpi cluster config when needed. You can also look into adding the config in the file instead of overriding it. More info here.

This tutorial is takes inspiration and guidance from Alex Ellis’s work. There are a few other tutorials and blog posts listed on Alex’s repo and are worth checking out for more k8s on rpi goodness.

This is a pretty decent cluster that one can use for learning purposes. With some volume mounted you can do some pretty cool stuff. I will try out some stuff in the upcoming days. Will also try to get the dashboard running.