Everything you need to know about the Kubernetes Raspberry Pi cluster
Last Updated: January 12, 2017
You may have heard of or seen the Kubernetes Raspberry Pi cluster. In this post I share everything I learned when I built one myself and demonstrated it live.
Hardware
You can find all the parts you need in Arjen Wassink’s Kubernetes Raspberry Pi Cluster Shopping List.
I highly recommend the ModMyPi stack case listed in Arjen’s shopping list, because the dimensions fit perfectly with the network switch and the USB power hub. I ordered the GeauxRobot Raspberry Pi 2 B 4-layer Dog Bone Stack because it had one-day delivery. However, the dimensions are too small and my cluster doesn’t look nearly as good :(
If you have access to a 3D printer, you can print a custom stack case too! Pance Cavkovski shared his 3D design on Twitter and Thingiverse:
Lastly, I also recommend a 2ft cable rather than a 1ft cable for easier access.
Software
Arjen posted the Kubernetes Raspberry Pi Installation Instructions — it’s easy to install and comes in a single package.
Recently, I started to use the kubernetes-on-arm project. It comes with pre-built Kubernetes images, but it also allows you to build the latest Kubernetes version.
There are also many other detailed resources, such as detailed instructions using ArchLinux and Kubernetes, how to compile Kubernetes from scratch for Raspberry Pi, and Cluster computing with Raspberry Pi with Kubernetes.
Networking
There are many strategies to connect your cluster to the Internet. I picked the simplest setup that routes all the traffic via my MacBook laptop.
In OS X, you can enable Internet Sharing; choose System Preferences > Sharing and select the Internet Sharing option. You should share the connection from Wi-Fi to computers using an Apple USB Ethernet Adapter.
However, I needed to assign a static IP address to each Raspberry Pi (especially the one that runs the Kubernetes master). I spent hours trying to figure this out. To save you time, this is the trick when using OS X!
- Get the Ethernet NIC MAC address of each Raspberry Pi
- In OS X, create a file named /etc/bootptab
- For each MAC address, add the following content
# Bootptab file
# Section 1 — ignored
%%
# Section 2 — used
# Hardware types: 1=Ethernet, 6=Wireless
# See http://www.ietf.org/rfc/rfc1700.txt
# hostname hwtype hwaddr ipaddr bootfile
rpi1 1 b8:27:eb:5b:0d:0f 192.168.2.101 boot
rpi2 1 b8:27:eb:a1:5f:5a 192.168.2.102 boot
.
.
.
rpiN 1 xx:xx:xx:xx:xx:xx 192.168.2.10x bootIf your Raspberry Pi already received a DHCP assigned IP address in the past, you’ll need to force release and refresh for a new IP address. From the Raspberry Pi, run:
$ sudo dhclient -rReducing Failure Detection Periods for Demonstration Purposes
In Kubernetes, the controller manager component will check node health periodically, and then evict the pods on unhealthy nodes after a timeout period. There are a couple of command line arguments you should be aware of:
--node-monitor-grace-period=40s: Amount of time which we allow running Node to be unresponsive before marking it unhealty. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status.
--node-monitor-period=5s: The period for syncing NodeStatus in NodeController.
--pod-eviction-timeout=5m0s: The grace period for deleting pods on failed nodes.To demonstrate how Kubernetes handles node failure in a live demo, I did not want to wait for 40-seconds to elapse before marking a node as unhealthy, nor can I wait for 5 minutes to see the pod being rescheduled. Thus, after installing Kubernetes on the Raspberry Pi, I needed to update the controller manager startup command line arguments.
If you used kubernetes-on-arm, you can update /etc/kubernetes/static-pods/master/master.json on the master node. Find the container configuration for controller-manager, and update the command attribute:
{
“name”: “controller-manager”,
“image”: “kubernetesonarm/hyperkube”,
“command”: [ “/hyperkube”, “controller-manager”,
“--master=127.0.0.1:8080”,
“--v=2”,
“--node-monitor-grace-period=10s”,
“--pod-eviction-timeout=5s”,
“--service-account-private-key-file=...",
“--root-ca-file=...” ],
“volumeMounts”:[...]
}Base Images
Most of the existing container images are built for the x86 architecture. You need to run ARM binaries/images on the Raspberry Pi. Luckily, Hypriot created several Raspberry Pi compatible Docker base images you can use to run applications written in: Java, Go, Node, Ruby, and more. They also have MySQL and Redis images as well. You can find many other Raspberry Pi compatible images by searching for image names containing “rpi-”.
Visualization
I believe that visualization is extremely important during a live demo of a complicated process. To help visualize Kubernete’s failure detection and pod rescheduling, I used a version of a Kubernetes visualizer that was originally developed by Brendan Burns.
The visualizer code and how to use it are on my GitHub:
Last but not least, pre-pull all of the images from all nodes for a responsive demo.
Great Resources
There are many people working with the Raspberry Pi Kubernetes cluster. Many have shared what they are working on, including:
- Using Ansible Playbook
- Running ELK Cluster
- Running GlusterFS
- This missing troubleshooting guide (Tuning parameters you don’t want to miss! Added on Aug 25, 2016)
- KubeCloud (“A small-scale, tangible cloud computing environment … for teaching and research purposes” Added on Aug 25, 2016)
- Sloop (Looks like the easiest way to install Kubernetes 1.3+, I haven’t had a chance to try. Added on Aug 26, 2016)
- Using Kubeadmn and Weave (One of the easiest ways I’ve seen so far. Added on January 12, 2017)
- Hypriot’s official way using kubeadmn and Flannel (Probably the most comprehensive guide, Added on January 12, 2017)
What’s next?
Scaling stateless workload is easy. But how do you schedule workload with persistent data? In the next part of this blog series, I’ll show you how I share a 250GB SSD as Kubernetes Persistent Volumes with LVM and NFS.
Last but not least, you can run Kubernetes in the cloud too! One of the easiest way to run a Kubernetes cluster is using Google Container Engine. Give it a try!
