Stories by Malcolm Jones on Medium

How I setup a Raspberry Pi 3 Cluster Using The New Docker Swarm Mode In 29 Minutes

Malcolm Jones — Sun, 10 Jul 2016 20:26:43 GMT

Helllooooo, everyone! I know it’s been a little while since I wrote about anything on here. I’ve been on vacation from work for the past week, and finally had the time to mess around with a couple of things I’ve wanted to work on for a while! Today I want to tell you about my journey to deploy a cluster management solution to four Raspberry Pi 3’s I had sitting around my apartment. If you have been hiding under a rock for the past year or two, open-source cluster management has become a huge thing in the tech world allowing teams to provide efficient resource isolation and sharing across distributed applications/frameworks. At work, I have been playing with Mesos (and Marathon) for a number of projects, so I took this as an opportunity, at first, to try out Kubernetes. There are a large number of great blog posts out there detailing how others managed to bring up their clusters from scratch. In particular, I looked at the following posts/projects:

A. https://github.com/luxas/kubernetes-on-arm
B. http://blog.kubernetes.io/2015/11/creating-a-Raspberry-Pi-cluster-running-Kubernetes-the-shopping-list-Part-1.html
C. http://blog.kubernetes.io/2015/12/creating-raspberry-pi-cluster-running.html
D. https://medium.com/google-cloud/everything-you-need-to-know-about-the-kubernetes-raspberry-pi-cluster-2a2413bfa0fa#.9k4t8rusk
E. https://github.com/Project31/ansible-kubernetes-openshift-pi3/tree/hypriot

Each solution is well documented and simple to follow, but I came across problems, most likely because I was attempting to use up-to-date hardware/software and NOT the old locked versions mentioned in the blog posts. Anyone who has played with these bleeding edge technologies knows that things can change drastically in a week, let alone a year. I wanted to use my RPi3’s (not the Raspberry Pi 2), and Hypriot v0.8.0 Barbossa (so many excellent new features and updated core packages). Also, Hypriot v0.8.0 Barbossa running on a Raspberry Pi 3 seems to be an incredibly powerful combination according to the community, and I did not want to miss out on any of that especially when I only have a week to hack on things before being swamped at work again. At some point, I touched base with @Quintus23M AKA Dieter Reuter (Docker Pirate at @HypriotTweets) and he told me about some of the magic the lovely people have been cooking up over at Docker with v1.12.0. The following is my experience setting up my cluster with the new Docker Swarm Mode!

My setup: Hardware

In case you want to follow my exact steps, here is my exact setup:

- 4 x Raspberry Pi 3 Model B
- 4 x Samsung EVO 32GB Class 10 Micro SDHC Card with Adapter (NOTE: 32GB not required, I just wanted extra space)
- 4 x Micro-USB cables
- 1 x Manhattan 7-Port USB 2.0 Ultra Hub
- 1 x NETGEAR 5-Port Gigabit Ethernet 10/100/1000Mbps Switch (GS205)
- 4 x Ethernet Cat-5 cables
- 1 x Sabrent Premium 3 Port Aluminum USB 3.0 Hub with Multi-In-1 Card Reader (12" cable)
- 1 x GeauxRobot Raspberry Pi 3 Model B 4-layer

5:36pm EDT: Preparation Step — Flash Micro SD Cards With Hypriot v0.8.0

First things first, I needed to download the Hypriot v0.8.0 image. Since I’m using OS X, I was able to leverage the Flash CLI tool developed by the Hypriot team (https://github.com/hypriot/flash). This little guy takes care of image customization very efficiently before the Raspberry Pi is even turned on. Here you can predefine a proper hostname, WIFI settings, and more. On top of that, it has error handling built into it, handles unmounting the SD card for writing, and prompts you when everything is done. Must have tool. For each SD card, I ran the command below with the following host names, scarlett-kub-master, scarlett-kub-slave1, scarlett-kub-slave2, scarlett-kub-slave3. (NOTE: Hostnames remain the same as when I was working on k8s).

# NOTE: ignore the $ signs, they represent your bash prompt

$ flash --hostname scarlett-kub-master --config device-init.yaml https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.0/hypriotos-rpi-v0.8.0.img.zip

Ready to go!

5:45pm EDT: Preparation Step — Set Up Password-less SSH Access

When dealing w/ multiple nodes and SSH Access, using SSH keys to connect to remote machines is a must. You do not want to have to type in your password every single time you interact with one of your remote servers; that’s madness. To set up passwordless authentication I did the following (NOTE: This assumes that you have a private key created already; if you don’t, see this guide on how to get setup http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/):

A. First, you need to obtain the IP addresses of each of the Raspberry Pi’s running on your LAN. If you’re savvy on the command line, you can use something like arp-scan. If you prefer a GUI, try LanScan Pro. I used Hypriot’s custom bash function to do a lookup against the Avahi-daemon configured on each Raspberry Pi:


# NOTE: Default password for the “pirate” user is “hypriot”

$ function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }

$ export MASTER=$(getip scarlett-kub-master.local)

$ echo $MASTER

$ export SLAVE_1=$(getip scarlett-kub-slave.local)

$ echo $SLAVE_1

$ export SLAVE_1=$(getip scarlett-kub-slave1.local)

$ echo $SLAVE_1

$ export SLAVE_2=$(getip scarlett-kub-slave2.local)

$ echo $SLAVE_2

$ export SLAVE_3=$(getip scarlett-kub-slave3.local)

$ echo $SLAVE_3

$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_1

$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_2

$ ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pirate@$SLAVE_3

B. Now to test that everything works, I ran:

$ ssh pirate@scarlett-kub-master.local

NICE!

5:47pm EDT: (Optional) Set up SSH Autocomplete

A. Open up ~/.ssh/config and add an entry that looks like the following:


HOST scarlett-kub-master
 HostName scarlett-kub-master.local
 Port 22
 User pirate
 IdentityFile ~/.ssh/id_rsa_ssh
 ServerAliveInterval 60
 ServerAliveCountMax 2
 ForwardX11 yes
 PasswordAuthentication no
 IdentitiesOnly yes
 LogLevel FATAL
 ForwardAgent yes

B. Add this line to your ~/.bash_profile then run source ~/.bash_profile:


# Add tab completion for SSH hostnames based on ~/.ssh/config, ignoring wildcards

[[ -e “$HOME/.ssh/config” ]] && complete -o “default” -o “nospace” -W “$(grep “^Host” ~/.ssh/config | grep -v “[?*]” | cut -d “ “ -f2 | tr ‘ ‘ ‘\n’)” scp sftp ssh

C. Reload your shell, type scarlett-kub-[TAB] and it should work. Repeat for each of the other RPi’s.

5:50pm EDT: Installation Of Docker 1.12.0-rc3 On Each Node

As of today, 7/8/2016 Docker 1.12.0 is a release candidate, rc3 to be exact. Luckily, @Quintus23M was able to hook me up with a custom compiled .deb package built from upstream! Here were the steps to install it:

A. Grab the binary then scp it to each machine:


$ wget https://jenkins.hypriot.com/job/armhf-docker/15/artifact/bundles/latest/build-deb/raspbian-jessie/docker-engine_1.12.0%7Erc3-0%7Ejessie_armhf.deb

$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-master.local:.

$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave1.local:.

$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave2.local:.

$ scp docker-engine_1.12.0~rc3–0~jessie_armhf.deb pirate@scarlett-kub-slave3.local:.

B. Remove the old docker-hypriot that is currently installed, and install rc3 (Repeat for each machine):


$ ssh pirate@scarlett-kub-master.local sudo apt-get purge -y docker-hypriot

$ ssh pirate@scarlett-kub-master.local sudo dpkg -i docker-engine_1.12.0~rc3–0~jessie_armhf.deb

C. SSH to machine, run docker version and you should see something like this:

LOOKING GOOD.

5:56pm EDT: Bootstrap The Master!

Believe it or not, this only takes one command. Run the following:


$ ssh pirate@scarlett-kub-master.local docker swarm init
Swarm initialized: current node (1njlvzi9rk2syv3xojw217o0g) is now a manager.

Yes …. That’s it. Verify that everything worked by running the following ( ssh to master node first ):

$ docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
al55dh1yjho2wojhagzksdqwu * scarlett-kub-master Accepted Ready Active Leader

Amazing.

5:58pm EDT: Time To Bootstrap Some Nodes!

Just when you thought it couldn’t get any easier… Run this from OS X( NOTE: Assume bash variable $MASTER is the ip address of your master node):

$ function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }

$ export MASTER=$(getip scarlett-kub-master.local)

$ ssh pirate@scarlett-kub-slave1.local docker swarm join $MASTER:2377

$ ssh pirate@scarlett-kub-slave2.local docker swarm join $MASTER:2377

$ ssh pirate@scarlett-kub-slave3.local docker swarm join $MASTER:2377

BOOM. Run that verification command again and you should see something like this:

Can’t believe it yet? Well, we are not done yet! Time to bring up some containers.

6:00pm EDT: Let’s bring up a container that simply pings google’s DNS server:

# run this from the master

$ docker service create --name ping hypriot/rpi-alpine-scratch ping 8.8.8.8
0ktotb58pbi0xp9op6anrdmkp

# verify the task was created

$ docker service tasks ping
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
706fyhabjxfxgtcwesobkkokz ping.1 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-master
Hypriot/armv7: pirate@scarlett-kub-master in ~
$

WOOT!

6:02pm EDT: Scale up the number of tasks running! How about 10 of these bad boys? No problem, let’s run this:

# update to 10

$ docker service update ping --replicas 10
ping
Hypriot/armv7: pirate@scarlett-kub-master in ~
$

# verify

$ docker service tasks ping
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
706fyhabjxfxgtcwesobkkokz ping.1 ping hypriot/rpi-alpine-scratch Running About a minute Running scarlett-kub-master
9lsto7r7qt1n2bua9ltirgll4 ping.2 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
4t0mpw823usq0jbm1lymh5ga2 ping.3 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
0vc51yzpe29qfzrz9uk3ffkx3 ping.4 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-master
2s01xvf4sjb7k558a7ga3b7hs ping.5 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave3
by7osvlr4wb7474brrlouv9ad ping.6 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave3
17uxtvgqa29hul6kuqs8penv3 ping.7 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
b6q27d3hmohr70f2odjuycyln ping.8 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
7rri99dki0fyj012cbenzi3o1 ping.9 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave2
1zmgn86bqtxj68ohxxo5226id ping.10 ping hypriot/rpi-alpine-scratch Running 14 seconds Running scarlett-kub-slave1
Hypriot/armv7: pirate@scarlett-kub-master in ~
$

# Alternative approach to scaling:

$ docker service scale ping=10
ping scaled to 10

Andddd, we’re done.

6:05pm EDT: Sit Back In Awe.

It’s that simple. Bravo to Hypriot and Docker. You guys just made my whole week! The craziest part is, this is my first time using HypriotOS and Docker together! 29Minutes start-to-finish. Madness.

Thanks for following the journey, please don’t hesitate to contact me or drop a comment if this was useful for you!

BAKING RACKSPACE PUBLIC CLOUD AND PRIVATE OPENSTACK IMAGES

Malcolm Jones — Wed, 05 Aug 2015 15:00:04 GMT

Hi friends!

Today’s blog post has to do with every sysadmin’s favorite task, baking images (insert sarcasm). In particular we’re going to talk about how to use Packer to bake images for Rackspace Public Cloud, and Private Openstack Cloud.

NOTE: Assume we are using Packer version v0.7.5

Baking Rackspace Public Cloud Images

The common issue I discovered with baking openstack images is that it is super easy to set MORE environment variables than needed. Setting too many can cause packer builds to fail with strange error messages. For example, as noted in this packer git issue:

source: https://github.com/mitchellh/packer/issues/1755
I’ve been trying with no success to use Packer to build an image with Rackspace. Currently here is my config:

{ 
    “variables”: { 
      “rackspace_username”: “”, 
      “rackspace_password”: “” 
  }, 
  “builders”: [ 
    { 
      “type”: “openstack”, 
      “username”: “{{user rackspace_username}}”, 
      “password”: “{{user rackspace_password}}”, 
      “provider”: “rackspace-us”, 
      “region”: “IAD”, 
      “ssh_username”: “root”, 
      “image_name”: “Test image {{timestamp}}”, 
      “source_image”: “ea98e34c-7a25–4670–9e2a-7fa9815f6c9f”,
      “flavor”: “2” 
    } 
  ], 
  “provisioners”: [ 
  { 
    “type”: “shell”, 
    “inline”: [ 
      “sleep 30”, 
      “sudo apt-get update”, 
      “sudo apt-get upgrade -y” 
      ] 
    } 
  ] 
}

But when I run:

packer build -var “rackspace_username=foo” -var “rackspace_password=bar” test.json

I get:

openstack output will be in this color.

Build ‘openstack’ errored: Auth failed. Bad credentials.

==> Some builds didn’t complete successfully and had errors: 
--> openstack: Auth failed. Bad credentials.
==> Builds finished but no artifacts were created.

Any ideas?

If you look closely at the packer documentation, you’ll notice that both the OS_* and SDK_* env variables have different “weights” associated with them, meaning if you set too many at the same time, they can override each other. Rule of thumb is SDK_* has more precedence than OS_*.

I put the following into a small script full of environment variable named local-public.sh with contents:

export SDK_USERNAME=”BLAH” 
export SDK_PASSWORD=’BLAH’ 
export SDK_API_KEY=’BLAH’ 
export SDK_PROVIDER=”rackspace-us” 
export SDK_REGION=”DFW”

NOTE: Notice for Rackspace Public Cloud I used only SDK_* environment variables.

Then in a new bash shell I ran source ./local-public.sh

Following that, I created a template that looks something like this:

{ 
“variables”: { 
  “SSH_PASSWORD”: “”, 
  “REGION”: “us-east-1”, 
  “INSTANCE_TYPE”: “m3.large”, 
  “SSH_USERNAME”: “centos”, 
  “SSH_TIMEOUT”: “5m” 
  }, 
  “builders”: [ 
    { 
      “type”: “openstack”, 
      “name”: “rackspace-public”, 
      “username”: “”, 
      “password”: “”, 
      “api_key”: “”, 
      “provider”: “rackspace-us”, 
      “openstack_provider”: “rackspace”, 
      “region”: “DFW”, 
      “ssh_username”: “root”, 
      “insecure”: “false”, 
      “image_name”: “centos66-{{timestamp}}”, 
      “source_image”: “blah”, 
      “flavor”: “performance1–1” 
    },

.....

NOTE: Notice I did not set any password/etc in the actual template. Packer reads in directly from your environment and populates everything accordingly.

Now packer build -only=rackspace-public openstack-centos66.json And boom! Good to go!

Baking Private Openstack Cloud

The steps for creating a private Openstack cloud image are very similar, but might require a little bit more information on your Openstack setup. Make sure you know how to query it for what you need!

I put the following into a small script named local-private.sh with contents:

# Source: https://www.packer.io/docs/builders/openstack.html

export OS_AUTH_URL=”http://OPENSTACK_AUTH_IP_ADDRESS:PORT/v2.0" export OS_USERNAME=”BLAH” 
export OS_PASSWORD=”BLAH” 
export OS_TENANT_ID=”BLAH”

# should be the same as OS_USERNAME
export OS_TENANT_NAME=”BLAH” 
export OS_AUTH_STRATEGY=”keystone”

NOTE: Notice for private openstack I used only OS_* environment variables.

Then in a new bash shell I ran source ./local-private.sh, make sure you don’t have any SDK_* variables set!!!!