A “production-ish” Kubernetes cluster on Raspberry Pi

Table of contents:

Let’s Build

  • Dynamic Persistent Volumes to add persistent data to pods
  • Externally accessible URLs
  • Load balancing of requests across multiple containers
  • Automatic backup and restore workflow
  • Easy installation of services via Helm
  • No out-of-band configuration of router or DNS records after initial setup
  • Auto-renewing TLS certificates from Let’s Encrypt
  • VPN access to cluster for debugging

Why though?

Hardware

  • Buy fewer Raspberry Pis (You’ll want at least 3 to avoid the cluster running out of memory)
  • Use ethernet cables, switches, etc you already have laying around (Most of the hardware above was picked because it’s the same physical size as the Raspberry Pi, but this is only an aesthetic choice)
  • Skip the case (just tape those Pis to a cardboard box, I won’t judge)
  • Use your current router rather than the Unifi networking equipment and 8-port switch (Your existing router will work fine for this setup with a couple small limitations, I’ll note later in the guide when the Unifi Router is required)

Networking

Draw it out

Example: deploy a VPN Service

  • ClusterIP: (default) Service is given a cluster-internal IP address and is accessible only to other apps within the cluster
  • NodePort: Service listens on a static port on the host machine, usually a high port like 30000
  • LoadBalancer: Service is exposed via some infrastructure load balancer service. Often this is a cloud-provider specific service like AWS ELBs
  • External IPs: Used in conjunction with any of the previous service types. Specifying an externalIP for a service will cause all worker nodes to start listening on that service's port. If a worker receives traffic on that port and the destination IP of that packet matches the externalIP of a service, the worker will route that packet to the service's pod via the kube-proxy process. Allows you to bind on low ports like 80 and 443 but requires a k8s user with elevated privileges
  • Choose ClusterIP if the service will only be accessed from within the cluster
  • Choose NodePort if the service should be externally accessible but you don’t have Load Balancer infrastructure in place and you don’t need a privileged port like 80 or 443
  • Choose ClusterIP with ExternalIP if the service should be externally accessible but you don’t have Load Balancer infrastructure in place and want a privileged port
  • Choose LoadBalancer if you want traffic balanced across all worker nodes containing pods for that service

BGP Mode (recommended)

Layer 2 Mode (use if your router doesn’t have BGP support)

Managing DNS records

Managing port forwarding rules

Handling HTTPS traffic

Generating TLS certificates

Storage

  • Volumes must be manually created prior to deploying the app
  • All pods that use the volume are locked to a single worker node
  • If the worker goes down the app will be inaccessible until the worker comes back

Installing software with Helm

helm install stable/openvpn
helm install --values openvpn-options.yml stable/openvpn

Initial Setup

Hardware Setup

  • Plug Unifi Gateway’s WAN1 port into your modem
  • Plug Unifi Gateway’s LAN1 port into the larger switch (note: the picture shows some components plugged into LAN2, we’ll cover this in a later section)
  • The Unifi AP should come with a Power-over-Ethernet (PoE) adapter
  • Plug the LAN port of the PoE adapter into the larger switch
  • Plug the PoE port of the PoE adapter into the Unifi AP
  • Assemble the Raspberry Pi’s into the stacking case (the case linked above includes a 7th level that we won’t use)
  • Plug USB cables into the charging dock but don’t plug then into the Raspberry Pi’s yet (we’ll power up the Pi’s after we flash the SD cards)
  • Plug the USB power cable of the mini switch into a USB port on one of the Pi’s
  • Plug each Pi into the mini switch using the mini ethernet cables
  • Plug the mini switch into the larger switch using another mini ethernet cable
  • Plug everything into AC power

Ansible Introduction

[all]
k8s-node1 ansible_host=192.168.1.100
k8s-node2 ansible_host=192.168.1.101
k8s-node3 ansible_host=192.168.1.102
[kube-master]
k8s-node1
[kube-node]
k8s-node2
k8s-node3
- hosts: all
roles:
- glusterfs-client
- hosts: gfs-cluster
roles:
- glusterfs-server

Create project

mkdir k8s-pi
cd k8s-pi
git init
cat << EOF > .gitignore
/secrets/
/tmp/
*.retry
EOF
mkdir submodules
git submodule add https://github.com/ljfranklin/k8s-pi.git ./submodules/k8s-pi

Flash HypriotOS onto SD cards

ssh-keygen -t rsa -b 4096 -C k8s -N '' -f ~/.ssh/id_rsa_k8s
ssh-add ~/.ssh/id_rsa_k8s
./submodules/k8s-pi/pi/provision.sh -d /dev/sda -n k8s-node1 -p "$(cat ~/.ssh/id_rsa_k8s.pub)" -i 192.168.1.100
./submodules/k8s-pi/pi/provision.sh -d /dev/sda -n k8s-node2 -p "$(cat ~/.ssh/id_rsa_k8s.pub)" -i 192.168.1.101

Boot Raspberry Pi’s

ssh k8s@192.168.1.100
ssh k8s@192.168.1.105 sudo wipefs -a /dev/sda # assumes USB drive has device ID /dev/sda

Installing k8s

cat << EOF > ansible.cfg
[defaults]
host_key_checking = False
remote_user = k8s
roles_path = submodules/k8s-pi/roles/
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
EOF
sudo pip install -r submodules/k8s-pi/requirements.txt
mkdir inventory
cat << EOF > inventory/hosts.ini
[all]
k8s-node1 ansible_host=192.168.1.100
k8s-node2 ansible_host=192.168.1.101
k8s-node3 ansible_host=192.168.1.102
k8s-node4 ansible_host=192.168.1.103
k8s-node5 ansible_host=192.168.1.104
k8s-node6 ansible_host=192.168.1.105
[kube-master]
k8s-node1
[kube-node]
k8s-node2
k8s-node3
k8s-node4
k8s-node5
k8s-node6
[gfs-cluster]
k8s-node6 volume_device=/dev/sda
EOF
cat << EOF > bootstrap.yml
- name: Include k8s-pi bootstrap tasks
import_playbook: submodules/k8s-pi/bootstrap.yml
EOF
cat << EOF > upgrade.yml
- name: Include k8s-pi upgrade tasks
import_playbook: submodules/k8s-pi/upgrade.yml
EOF
cat << EOF > deploy.yml
- name: Include k8s-pi deploy tasks
import_playbook: submodules/k8s-pi/deploy.yml
EOF
mkdir -p secrets
cp submodules/k8s-pi/secrets/secrets.sample secrets/secrets.yml
ansible-playbook -i inventory/hosts.ini --extra-vars @secrets/secrets.yml bootstrap.yml
cp ./secrets/admin.conf ~/.kube/config

Optional: Setup Unifi Controller

  • Ensure all devices are plugged into LAN1 port of Unifi Gateway (we’ll switch some devices over to LAN2 in a later step to enable BGP routing)
  • Temporarily modify /etc/hosts on your workstation to route Controller traffic directly to the first worker node: sudo vim /etc/hosts to add $FIRST_WORKER_NODE_IP unifi.$INGRESS_DOMAIN Note: We’ll revert this change once the Unifi controller is setup with BGP routing
  • Visit https://unifi.$INGRESS_DOMAIN
  • Configure Wifi network name/password and controller/device username/passwords
  • Adopt the Unifi Gateway (detailed steps here):
  • > ssh ubnt@192.168.1.1 (password ubnt)
  • > If gateway was previously paired: sudo syswrapper.sh restore-default, SSH session may get stuck, may need to kill it and re-SSH after reboot
  • > set-inform http://$FIRST_WORKER_IP:8080/inform
  • > Go to Controller UI and click Adopt on Devices tab
  • > Wait for device to go from Adopting to Provisioning to Connected on Controller UI
  • Adopt the Unifi AP:
  • > If the AP was not previously paired with another controller: The AP should appear in the Devices tab, click Adopt next to it in the UI
  • > If the AP was previously paired: Hold down the small Reset button on the AP with a paper clip to reset to factory default settings
  • > After reseting the AP should appear in the Devices tab, click Adopt
  • Add temporary port forwarding rule to bootstrap port-forwarding-controller:
  • > Settings > Routing & Firewall > Port Forwarding > Create New Port Forwarding Rule
  • > Name: tmp-k8s-ingress
  • > Port: 443
  • > Forward IP: $ingressnginxstatic_ip (from secrets.yml)
  • > Forward Port: 443
  • > Save
  • Restart the port-forwarding-controller to ensure it adds port forwarding rules to controller
  • > kubectl delete pod port-forwarding-0
  • Verify rules were added under Settings > Routing & Firewall > Port Forwarding
  • Delete tmp-k8s-ingress rule
  • Remove $WORKER_IP line from /etc/hosts
  • Done!

Optional: Move non-k8s machines over to LAN2 to enable BGP routing

  • Start with all machines plugged into LAN1 port on Unifi Gateway
  • Visit https://unifi.$INGRESS_DOMAIN
  • Enable LAN2 network:
  • > Settings > Networks > Create New Network
  • > Name: LAN2
  • > Interface: LAN2
  • > Gateway/Subnet: 192.168.2.1/24
  • > DHCP Range: 192.168.2.6–192.168.2.254
  • > All other values default
  • > Save
  • Unplug the mini switch from the larger switch
  • Unplug the larger switch from the router’s LAN1 port and plug in into LAN2
  • Plug the mini switch into the LAN2 port
  • Connect all other machines (desktop, Wifi AP, etc.) into LAN2 switch
  • Verify that you can now route directly to BGP LB addresses:
  • > nc -v -z $ingress_nginx_static_ip 443
  • > Should say Connection to 192.168.1.51 443 port [tcp/https] succeeded! or similar

Optional: Access K8S Dashboard

Optional: Connect to cluster with VPN

./submodules/k8s-pi/scripts/generate-vpn-cert.sh vpn.$INGRESS_DOMAIN

Optional: Backup/Restore

Take backup on-demand

ark create backup backup-01032019 --ttl 360h0m0s
ark describe backup backup-01032019 --volume-details
ark get backups

Restoring entire cluster from backup

$ ark get backups
NAME STATUS CREATED EXPIRES SELECTOR
daily-backups-20190103070021 Completed 2019-01-02 23:00:21 -0800 PST 13d <none>
daily-backups-20190102070021 Completed 2019-01-01 23:00:21 -0800 PST 12d <none>
...
ark create restore restore-01032019 --from-backup backup daily-backups-20190103070021
ark describe restore restore-01032019 --volume-details

Restoring select deployments from backup

ark create restore restore-01032019 --from-backup backup daily-backups-20190103070021 --label app=openvpn

Possible gotcha: Restic Repo shows NotReady

kubectl -n heptio-ark exec -it ark-restic-POD_ID /bin/sh
restic unlock -r gs:<VOLUME_BACKUP_BUCKET>:default # enter 'static-passw0rd' as the repo password

Optional: Adding your own Ansible tasks

Optional: Building ARM images

  • Check whether the image maintainers created the arm image under a different name or tag, e.g. k8s.gcr.io/defaultbackend-arm:1.4
  • Google around to see whether someone else has uploaded an arm image to Dockerhub for that piece of software
  • > Disclaimer: use at your own risk when using software from someone other than the official project maintainers
  • Build the image yourself on a Raspberry Pi, here are the usual steps:
  • > clone the target project from GitHub onto the Raspberry Pi
  • > cd to the directory containing the Dockerfile
  • > Run docker build -t YOUR_USERNAME/IMAGE-arm:latest .
  • > Run docker login and paste in your Dockerhub credentials
  • > Run docker push YOUR_USERNAME/IMAGE-arm:latest
  • > Specify the new image name and tag in the helm values file, the name of the key will vary by chart
  • Open an issue asking the project maintainers to start publishing arm images

Open Issues

Future work

  • Switch to Kubespray for base Ansible playbooks
  • > Like this project, Kubespray is a collection of Ansible roles to setup a k8s cluster, but doesn’t support ARM out-of-the-box
  • > I tried for ~1 day to get Kubespray working but I got stuck on GlusterFS volumes hanging on mount
  • Support multi-master deployment of master node and etcd
  • > The above setup only has a single master k8s node
  • > If this node goes down you’ll be unable to modify any cluster resources, luckily the workers can continue to serve application traffic

Finished!

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store