This was written over 20 months ago. Which is a lifetime in cloud native years. Many things probably does not work as described here. Please find more up-to-date tutorials if this topic is still of interest. I hope to rebuild this again with the new 2 core 8 Gb ram pis that is available now.
Over the winter break I had some free time to deploy a Kubernetes cluster using raspberry pis. The documentation around this is quite good but I wanted to make a super beginner guide on this.
What you need (must haves)
- Raspberry PI (2/3). Minimum of 2 (1 master and 1 worker)
- Power Cable (Micro USB, same as the old android chargers)
- Ethernet Cable (Can be done with wireless, but ethernet cable makes it easier)
- Micro SD Card (Minimum of 8gb)
What you want (Nice to have)
- Raspberry PI 3 Model b+. 4 (1 master and 3 worker)
- Gigabit Ethernet Switch.
- Short (1 ft) ethernet cable
- Short (1 ft) Power cable
- Multiport USB Charging HUB. (I used a 6 port Anker hub)
- Cluster Case (Keeps everything nice and clean)
- Micro SD Card (32gb each)
- Raspbian Stretch Lite
- Tmux (Useful for sending command to multiple terminal windows)
- Access to the Router Admin or IPScanner software.
- I am following this tutorial to setup. If anything I do is confusing or too slow for your pace feel free to jump there. My goal is to make this version beginner-friendly.
- Using etcher (or something similar) burn the os image onto the MicroSD card. You can follow this guide for instruction.
- Put an empty file named ‘ssh’ (the file has no file extension or any content in) inside the SD card. This will allow us to ssh into the device out of the box. That way we skip the need for connecting a monitor and other input peripherals.
- Insert the SD cards in the raspberry pi and power on the pis. If you check in your router connected device you should see a few new devices pop up. The hostname on all of these would be raspberry. If you can’t see the devices on your router you can use a IPScanner, I used Angry IP Scanner.
- Since there are multiple devices that need some setting up, Tmux will come in super handy. Although this can totally be done without tmux. But I had 4 pis to setup and doing the same thing 4 times is not my thing. If you never used tmux you are in for a treat (or torture, or maybe both). This guide will get you started. I will use the synchronize-pane feature and turn it on and off as I need.
- At this point, I am hoping you are ssh’ed in all the pis. Type in
raspi-config. It is a built in raspberry pi tool to configure the device.
7. You should see something similar. Go to Network Options > Hostname. And change the hostname to anything you want. I named mine k8s-master-1, k8s-node-1, k8s-node-2 and k8s-node-3.
8. Once you are done with that, you should set up Localization Options. Although it is optional for our use-case. It is just good practice. Also without it you won’t be able to enable wifi.
9. Once done the pi will reboot.
10. ssh in again. We will give our device a static ip so it keeps the ip between reboots.
cat >> /etc/dhcpcd.conf
Find your device IP. It will be in the form of x.x.x.x. You can find your device ip (on a mac) using
ifconfig | grep inet . My mac ip was
192.168.0.26 so my ip format is like
Paste the following code block
Where x.x.x is same as your device ip and y is the ip you want. I put 100 for master and 101,102 and 103 for the rest. Reboot with
When the devices turn on they should reflect the new IPs.
This concludes the initial setup. Next we do the general device setup.
This bit of setup is common for all the raspberry pis. Using sync-pane from tmux is perfect here.
- The main choices for a container environment are Docker and cri-o. We will user Docker, as cri-o requires a fair amount of extra work to enable for Kubernetes. As cri-o is open source the community seems to be heading towards its use The following command installs docker and sets the right permission.
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker && \
2. We need to then disable swap. Kubernetes requires swap to be disabled. More info about it on this pull request.
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove
We can check swap disable was a success by the following command returning empty
sudo swapon --summary
3. Next we edit the
/boot/cmdline.txt file. Add the following in the end of the file. This needs to be in the same line as all the other text in the file. Do not create a new file.
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
4. Reboot with
5. SSH in again. Edit the following file
add the following in the file
deb http://apt.kubernetes.io/ kubernetes-xenial main
Add the key
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
If it works, it will output
6. Update with new repo, which will download new repo information.
sudo apt-get update
kubeadm it will also install
sudo apt-get install -qy kubeadm
This concludes the common setup.
Master Node Setup
This bit of setup is just for the master node.
- Pre-pull images
sudo kubeadm config images pull -v3
2. We will be using Weave Net as a network overlay.
sudo kubeadm init --token-ttl=0
- -token-ttl = 0 makes sure our token doesn’t expire. This is not a good practice and should not be done in production.
3. This step takes a long time. (15 mins almost). Once its done it will give a snippet of code to run. Run it to set up
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
4. You should also see the join-token. You need this token for other nodes to join your network. It will look something like this
kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea
5. Incase you loose this line or can’t find in the terminal, you can find the token in the following way.
kubeadm token list will show you all available token. From there you can find the token. Finding the
sha hash is a bit trickier. But you can get it by
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
This long and convoluted command gets the
sha256 for the certificate.
6. Install the Weave Net network driver
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
7. Check Everything worked.
NAMESPACE NAME READY STATUS
kube-system coredns-86c58d9df4-9hx5c 1/1 Running
kube-system coredns-86c58d9df4-nfgk5 1/1 Running
kube-system etcd-k8s-master-1 1/1 Running
kube-system kube-apiserver-k8s-master-1 1/1 Running
kube-system kube-controller-manager-k8s-master-1 1/1 Running
kube-system kube-proxy-4k2mc 1/1 Running
kube-system kube-scheduler-k8s-master-1 1/1 Running
kube-system weave-net-5rmmn 2/2 Running
You want the status to be running. If its not, something went wrong. (If its pending or container creating give it some time). But if you see it stuck for too long (3–4 mins) trace your steps back and reconfigure.
8. On the master and all the workers run the following command
sudo sysctl net.bridge.bridge-nf-call-iptables=1
Reasoning for this is explained better in this issue
Worker Node Setup
- From each worker node run
sudo kubeadm join --token <token> <master-node-ip>:6443 --discovery-token-ca-cert-hash sha256:<sha256>
2. After a few moments, run
kubectl get nodes
and you should see something like
NAME STATUS ROLES AGE VERSION
k8s-master-1 Ready master 11h v1.13.1
k8s-node-1 Ready worker 11h v1.13.1
k8s-node-2 Ready worker 11h v1.13.1
k8s-node-3 Ready worker 11h v1.13.1
And with this the cluster setup is done.
We can access our cluster now, which is awesome. But we can only do it while ssh’ed in the master node. We should change that.
We can copy the config from the master to our local machine. Use
scp . More info on this on this article.
scp email@example.com:.kube/config .
Running the command from you local directory will copy the
config file in that directory from master-node.
If you have
kubectl installed in you local machine, you can then setup
kubeconfig by either overriding the
config file in
$HOME/.kube/config or you can add the file on top of the existing config (if you have it) by
export KUBECONFIG=<location to config from pi>:$HOME/.kube/config
This way you can keep you
kubeconfig and switch to rpi cluster config when needed. You can also look into adding the config in the file instead of overriding it. More info here.
This tutorial is takes inspiration and guidance from Alex Ellis’s work. There are a few other tutorials and blog posts listed on Alex’s repo and are worth checking out for more k8s on rpi goodness.
This is a pretty decent cluster that one can use for learning purposes. With some volume mounted you can do some pretty cool stuff. I will try out some stuff in the upcoming days. Will also try to get the dashboard running.