How I setup a Raspberry Pi 3 Cluster Using The New Docker Swarm Mode In 29 Minutes

8 min readJul 10, 2016

Helllooooo, everyone! I know it’s been a little while since I wrote about anything on here. I’ve been on vacation from work for the past week, and finally had the time to mess around with a couple of things I’ve wanted to work on for a while! Today I want to tell you about my journey to deploy a cluster management solution to four Raspberry Pi 3’s I had sitting around my apartment. If you have been hiding under a rock for the past year or two, open-source cluster management has become a huge thing in the tech world allowing teams to provide efficient resource isolation and sharing across distributed applications/frameworks. At work, I have been playing with Mesos (and Marathon) for a number of projects, so I took this as an opportunity, at first, to try out Kubernetes. There are a large number of great blog posts out there detailing how others managed to bring up their clusters from scratch. In particular, I looked at the following posts/projects:

A. https://github.com/luxas/kubernetes-on-arm
B. http://blog.kubernetes.io/2015/11/creating-a-Raspberry-Pi-cluster-running-Kubernetes-the-shopping-list-Part-1.html
C. http://blog.kubernetes.io/2015/12/creating-raspberry-pi-cluster-running.html
D. https://medium.com/google-cloud/everything-you-need-to-know-about-the-kubernetes-raspberry-pi-cluster-2a2413bfa0fa#.9k4t8rusk
E. https://github.com/Project31/ansible-kubernetes-openshift-pi3/tree/hypriot

Each solution is well documented and simple to follow, but I came across problems, most likely because I was attempting to use up-to-date hardware/software and NOT the old locked versions mentioned in the blog posts. Anyone who has played with these bleeding edge technologies knows that things can change drastically in a week, let alone a year. I wanted to use my RPi3’s (not the Raspberry Pi 2), and Hypriot v0.8.0 Barbossa (so many excellent new features and updated core packages). Also, Hypriot v0.8.0 Barbossa running on a Raspberry Pi 3 seems to be an incredibly powerful combination according to the community, and I did not want to miss out on any of that especially when I only have a week to hack on things before being swamped at work again. At some point, I touched base with @Quintus23M AKA Dieter Reuter (Docker Pirate at @HypriotTweets) and he told me about some of the magic the lovely people have been cooking up over at Docker with v1.12.0. The following is my experience setting up my cluster with the new Docker Swarm Mode!

My setup: Hardware

In case you want to follow my exact steps, here is my exact setup:

- 4 x Raspberry Pi 3 Model B
- 4 x Samsung EVO 32GB Class 10 Micro SDHC Card with Adapter (NOTE: 32GB not required, I just wanted extra space)
- 4 x Micro-USB cables
- 1 x Manhattan 7-Port USB 2.0 Ultra Hub
- 1 x NETGEAR 5-Port Gigabit Ethernet 10/100/1000Mbps Switch (GS205)
- 4 x Ethernet Cat-5 cables
- 1 x Sabrent Premium 3 Port Aluminum USB 3.0 Hub with Multi-In-1 Card Reader (12" cable)
- 1 x GeauxRobot Raspberry Pi 3 Model B 4-layer

5:36pm EDT: Preparation Step — Flash Micro SD Cards With Hypriot v0.8.0

First things first, I needed to download the Hypriot v0.8.0 image. Since I’m using OS X, I was able to leverage the Flash CLI tool developed by the Hypriot team (https://github.com/hypriot/flash). This little guy takes care of image customization very efficiently before the Raspberry Pi is even turned on. Here you can predefine a proper hostname, WIFI settings, and more. On top of that, it has error handling built into it, handles unmounting the SD card for writing, and prompts you when everything is done. Must have tool. For each SD card, I ran the command below with the following host names, scarlett-kub-master, scarlett-kub-slave1, scarlett-kub-slave2, scarlett-kub-slave3. (NOTE: Hostnames remain the same as when I was working on k8s).

# NOTE: ignore the $ signs, they represent your bash prompt$ flash --hostname scarlett-kub-master --config device-init.yaml https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.0/hypriotos-rpi-v0.8.0.img.zip

Ready to go!

5:45pm EDT: Preparation Step — Set Up Password-less SSH Access

When dealing w/ multiple nodes and SSH Access, using SSH keys to connect to remote machines is a must. You do not want to have to type in your password every single time you interact with one of your remote servers; that’s madness. To set up passwordless authentication I did the following (NOTE: This assumes that you have a private key created already; if you don’t, see this guide on how to get setup http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/):

A. First, you need to obtain the IP addresses of each of the Raspberry Pi’s running on your LAN. If you’re savvy on the command line, you can use something like arp-scan. If you prefer a GUI, try LanScan Pro. I used Hypriot’s custom bash function to do a lookup against the Avahi-daemon configured on each Raspberry Pi:


# NOTE: Default password for the “pirate” user is “hypriot”$ function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }$ export MASTER=$(getip scarlett-kub-master.local)$ echo $MASTER$ export SLAVE_1=$(getip scarlett-kub-slave.local)$ echo $SLAVE_1$ export SLAVE_1=$(getip scarlett-kub-slave1.local)$ echo $SLAVE_1$ export SLAVE_2=$(getip scarlett-kub-slave2.local)$ echo $SLAVE_2$ export SLAVE_3=$(getip scarlett-kub-slave3.local)$ echo $SLAVE_3$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_1$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_2$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_3

B. Now to test that everything works, I ran:

$ ssh pirate@scarlett-kub-master.local

NICE!

5:47pm EDT: (Optional) Set up SSH Autocomplete

A. Open up ~/.ssh/config and add an entry that looks like the following:


HOST scarlett-kub-master
 HostName scarlett-kub-master.local
 Port 22
 User pirate
 IdentityFile ~/.ssh/id_rsa_ssh
 ServerAliveInterval 60
 ServerAliveCountMax 2
 ForwardX11 yes
 PasswordAuthentication no
 IdentitiesOnly yes
 LogLevel FATAL
 ForwardAgent yes

B. Add this line to your ~/.bash_profile then run source ~/.bash_profile:


# Add tab completion for SSH hostnames based on ~/.ssh/config, ignoring wildcards[[ -e “$HOME/.ssh/config” ]] && complete -o “default” -o “nospace” -W “$(grep “^Host” ~/.ssh/config | grep -v “[?*]” | cut -d “ “ -f2 | tr ‘ ‘ ‘\n’)” scp sftp ssh

C. Reload your shell, type scarlett-kub-[TAB] and it should work. Repeat for each of the other RPi’s.

5:50pm EDT: Installation Of Docker 1.12.0-rc3 On Each Node

As of today, 7/8/2016 Docker 1.12.0 is a release candidate, rc3 to be exact. Luckily, @Quintus23M was able to hook me up with a custom compiled .deb package built from upstream! Here were the steps to install it:

A. Grab the binary then scp it to each machine:


$ wget https://jenkins.hypriot.com/job/armhf-docker/15/artifact/bundles/latest/build-deb/raspbian-jessie/docker-engine_1.12.0%7Erc3-0%7Ejessie_armhf.deb$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-master.local:.$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave1.local:.$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave2.local:.$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave3.local:.

B. Remove the old docker-hypriot that is currently installed, and install rc3 (Repeat for each machine):


$ ssh pirate@scarlett-kub-master.local sudo apt-get purge -y docker-hypriot$ ssh pirate@scarlett-kub-master.local sudo dpkg -i docker-engine_1.12.0~rc3–0~jessie_armhf.deb

C. SSH to machine, run docker version and you should see something like this:

LOOKING GOOD.

5:56pm EDT: Bootstrap The Master!

Believe it or not, this only takes one command. Run the following:


$ ssh pirate@scarlett-kub-master.local docker swarm init
Swarm initialized: current node (1njlvzi9rk2syv3xojw217o0g) is now a manager.

Yes …. That’s it. Verify that everything worked by running the following ( ssh to master node first ):

$ docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
al55dh1yjho2wojhagzksdqwu * scarlett-kub-master Accepted Ready Active Leader

Amazing.

5:58pm EDT: Time To Bootstrap Some Nodes!

Just when you thought it couldn’t get any easier… Run this from OS X( NOTE: Assume bash variable $MASTER is the ip address of your master node):

$ function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }$ export MASTER=$(getip scarlett-kub-master.local)$ ssh pirate@scarlett-kub-slave1.local docker swarm join $MASTER:2377$ ssh pirate@scarlett-kub-slave2.local docker swarm join $MASTER:2377$ ssh pirate@scarlett-kub-slave3.local docker swarm join $MASTER:2377

BOOM. Run that verification command again and you should see something like this:

Can’t believe it yet? Well, we are not done yet! Time to bring up some containers.

6:00pm EDT: Let’s bring up a container that simply pings google’s DNS server:

# run this from the master$ docker service create --name ping hypriot/rpi-alpine-scratch ping 8.8.8.8
0ktotb58pbi0xp9op6anrdmkp# verify the task was created$ docker service tasks ping
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
706fyhabjxfxgtcwesobkkokz ping.1 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-master
Hypriot/armv7: pirate@scarlett-kub-master in ~
$

WOOT!

6:02pm EDT: Scale up the number of tasks running! How about 10 of these bad boys? No problem, let’s run this:

# update to 10$ docker service update ping --replicas 10
ping
Hypriot/armv7: pirate@scarlett-kub-master in ~
$# verify$ docker service tasks ping
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
706fyhabjxfxgtcwesobkkokz ping.1 ping hypriot/rpi-alpine-scratch Running About a minute Running scarlett-kub-master
9lsto7r7qt1n2bua9ltirgll4 ping.2 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
4t0mpw823usq0jbm1lymh5ga2 ping.3 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
0vc51yzpe29qfzrz9uk3ffkx3 ping.4 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-master
2s01xvf4sjb7k558a7ga3b7hs ping.5 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave3
by7osvlr4wb7474brrlouv9ad ping.6 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave3
17uxtvgqa29hul6kuqs8penv3 ping.7 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
b6q27d3hmohr70f2odjuycyln ping.8 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
7rri99dki0fyj012cbenzi3o1 ping.9 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
1zmgn86bqtxj68ohxxo5226id ping.10 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
Hypriot/armv7: pirate@scarlett-kub-master in ~
$# Alternative approach to scaling:$ docker service scale ping=10
ping scaled to 10

Andddd, we’re done.

6:05pm EDT: Sit Back In Awe.

It’s that simple. Bravo to Hypriot and Docker. You guys just made my whole week! The craziest part is, this is my first time using HypriotOS and Docker together! 29Minutes start-to-finish. Madness.

Thanks for following the journey, please don’t hesitate to contact me or drop a comment if this was useful for you!