Running swarm mode on an ARM / x64 hybrid cluster

Since a couple of weeks, I really enjoy playing with Docker on ARM devices and especially testing swarm mode on little clusters. Below is an example of local cluster using swarm mode and made up of 2 PI2 and 1 PI3 (PI zero and CHIP should be joining the swarm party soon). A service running 5 replicas of alexellis2/arm-hellonode image is deployed.

Note: I refer to it as a local cluster as all the nodes are within the same home LAN.

For more information on how to play with Docker on ARM, give a look to Alex Ellis blog, lot of interesting things there.

Swarm mode in 30 secs

Basically, swarm mode is one of the great feature of Docker 1.12, this is an optional mode of the Docker engine that enables to create a secure cluster very easily.

Check Docker documentation and start playing (yes, you’ll have some fun !).

Swarm mode overview

Why would we need an hybrid cluster ?

First of all, what do I mean by hybrid cluster ? Simply a cluster that contains both nodes running on ARM and nodes running on x64–86 (x64) architectures.

Many applications needs servers with lots of RAM and CPU to correctly run bunch of (micro)services: DB, MessageQueue, web servers, workers, heavy Java applications…

Due to their technical specifications, most ARM devices are not the best fit to run such applications. But those devices are so great because they can act as bridges between computing and the real world.

For instance, it’s easy to have a raspberry blinking a light when he detects a new tweet containing the #docker hashtag. The possibility to add to today’s applications the capability to interact with the world is what motivates this article about an hybrid cluster.

We can consider several use-cases (fun and more useful) as examples of capability that could be added to existing field specific applications

• alert user by flashing home lights when someone select his tinder profile (no comment)
• add photo shooting capability of stars constellations at several locations at the same time
• add demand-response capability to energy specific application by physically switching off dedicated electric equipments

Let’s setup a demo hybrid cluster

We’ll use an ARM device (PI3) and two x64 machines.

• manager1 running on a PI3
• worker1 running on a DigitalOcean instance
• worker2 running on an Amazon EC2 instance

manager1 is running a Raspbian NOOBS 1.9.2 distribution and Docker 1.12.1 has been installed with the following command.

curl -sSL https://get.docker.com | sh

We’ll use Docker Machine to create worker1 and worker2. worker1 is a DigitalOcean droplet running Ubuntu 16.04, worker2 is a AWS EC2 instance running Ubuntu 14.04.

$ docker-machine create \
--driver digitalocean \
--digitalocean-access-token=$TOKEN \
--digitalocean-region=lon1 \
--digitalocean-size=1gb \
--digitalocean-image=ubuntu-16–04-x64 \
worker1

docker-machine create \
--driver amazonec2 \
--amazonec2-access-key=${ACCESS_KEY_ID} \
--amazonec2-secret-key=${SECRET_ACCESS_KEY} \
--amazonec2-region=eu-west-1 \
--amazonec2-ami=ami-ed82e39e \
--amazonec2-instance-type=t2.medium \
worker2

We now have our 3 nodes created (each one running Docker 1.12.1). Let’s initiate the swarm from manager1 and add both workers to the party.

Init the swarm from manager1 (raspberry pi3)

pi@manager1:~ $ docker swarm init --advertise-addr MANAGER_PUBLIC_IP --listen-addr PRIVATE_IP:2377
Swarm initialized: current node (aq7cy9pijja19v77cv2ppjb8l) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join \
--token SWMTKN-1–1l4igwytvbuyuo7ij55epiu9ywpdky5tlaqfy2jholeek93io5–8mt6okwcyy07tztpvuvi31b90 \
MANAGER_PUBLIC_IP:2377

To add a manager to this swarm, run ‘docker swarm join-token manager’ and follow the instructions.

After having initialized the cluster on manager1, we are provided commands to add additional workers and managers. In the current example, we’ll add 2 workers.

Note: in 1.12 it’s not possible to add several nodes (workers or managers) behind the same home router as in order to communicate each node of the swarm uses ports 2377, 7946 and 4789. The router needs to open those ports and use NAT to target only one node.

ssh to worker1 and join the swarm

root@worker1:~# docker swarm join --token SWMTKN-1–1l4igwytvbuyuo7ij55epiu9ywpdky5tlaqfy2jholeek93io5–8mt6okwcyy07tztpvuvi31b90 MANAGER_PUBLIC_IP:2377
This node joined a swarm as a worker.

ssh to worker2 and join the swarm

ubuntu@worker2:~$ sudo docker swarm join --token SWMTKN-1–1l4igwytvbuyuo7ij55epiu9ywpdky5tlaqfy2jholeek93io5–8mt6okwcyy07tztpvuvi31b90 MANAGER_PUBLIC_IP:2377
This node joined a swarm as a worker.

Let’s check the cluster status using the node api on the manager node. The output below shows that our 3 nodes swarm cluster is running correctly.

Let’s run our first service

From the manager, let’s create a web service with 6 replicas of nginx.

The output above shows that the 6 replicas have been scheduled on the 3 nodes (2 per nodes) but the 2 tasks scheduled on manager1 keep on failing. As the nginx image is compiled for x64 architecture, it thus cannot be ran on ARM.

Let’s remove this service (docker service rm web) and create another one using an image compiled for ARM.

Among the 6 replicas (2 per nodes) only the 2 tasks scheduled on manager1 are running fine. As the image used by the service is compiled for ARM architecture, this explains it cannot be deployed on x64.

Add labels on nodes

To schedule services using ARM images on ARM based nodes and x64 images on x64 nodes, we can specify labels on nodes and use those labels as constraints when running the services. Let’s see that in action.

The following commands set arc=arm to the manager’s node and arc=x64 label to worker1 and worker2.

Let’s now recreate the web service indicating it must run on x64 nodes.

Everything is fine this time as all the tasks are scheduled on worker1 and worker2.

Let’s recreating the hello service indicating it must run on node flagged with arc=arm label. As we see below, tasks are only scheduled on manager1, the only node with label arc=arm.

Summary

This example shows that it is possible to create a ARM / x64 hybrid swarm cluster. Such a cluster can be used to deploy services based on ARM images and services based on x64 images. Services running on one architecture should only be scheduled on corresponding nodes, this can be done using node’s labels and service’s constraints. ARM services and x64 services will then be able to run altogether in the swarm and communicate with each other. We’ll setup a real world example in a future article.

Where to go from here ?

Let’s think about the real world extensions we could add to the web applications we use today. Some of them are probably good fit and could be extended with fun and/or useful features. ARM devices open a lot of possibilities to interact with users through the usage of many physical sensors.

Note: in the current Docker version, the service api available in swarm mode does not enables tasks to run with --privileged or additional seccomp capability (like SYS_RAWIO) that would provide access to the physical peripheries of the devices. Hopefully service will soon be able to run with those additional options.

Special thanks to Alex Ellis and Hypriot for they really great ARM related articles and other great stuff.