How to run a command on every node in a Docker Swarm cluster

Iain Collins
Nov 15, 2017 · 3 min read

Docker Swarm is a great way to deploy Docker Containers on AWS or Azure, both of which are supported cloud providers for Docker Community Edition (CE) and Enterprise Edition (EE).

A common admin task can be needing to run a command on all nodes on a cluster, either for monitoring or when debugging problems.

Executing Containers on all nodes in a cluster is easy, and the docker stats command lets you easily view the memory, CPU, network and IO usage of all Containers on an individual node.

However, unless you are running the Enterprise Edition or something like the Portainer management UI, there is easy way to see the status of all containers at once — and there is no built-in docker command for running commands directly on nodes.

Combine docker, cut and xargs

If you have a Docker Swarm cluster you will already be familiar with using the docker node command to list nodes in the cluster:

> docker node ls
ID HOSTNAME STATUS
2v77j8x4ljejeeku9tao00tcw * swarm-manager000000 Ready ox633ovtcp5ux986fb30x6d6x swarm-manager000001 Ready
qfig0i98hhxdekf7wfld8jo7q swarm-worker000000 Ready

If you want to run commands on all nodes in a cluster, it’s pretty easy to combine this output with xargs using cut and ssh.

Examples

These examples assume you have SSH access to nodes already setup — this is the default behaviour after a Docker Swarm cluster has been setup with Docker Community Edition or Docker Enterprise Edition — if you can’t access all the nodes via SSH when logged in to the cluster you should resolve that first.

Checking uptime and load on all nodes

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER uptime"swarm-manager000000
03:21:23 up 42 days, 6:26, load average: 0.22, 0.14, 0.10
swarm-manager000001
03:33:42 up 41 days, 14:27, load average: 0.00, 0.02, 0.00
swarm-worker000000
03:33:42 up 40 days, 9:33, load average: 0.05, 0.03, 0.00

Checking free memory (in MB) on all nodes

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER free -m"swarm-manager000000
total used free shared buffers cached
Mem: 6956 5068 1888 162 182 1329
-/+ buffers/cache: 3555 3400
Swap: 0 0 0
swarm-manager000001
total used free shared buffers cached
Mem: 6956 6158 797 162 2320 2301
-/+ buffers/cache: 1537 5419
Swap: 0 0 0
swarm-worker000000
total used free shared buffers cached
Mem: 6956 6075 880 162 1956 2636
-/+ buffers/cache: 1482 5473
Swap: 0 0 0

Get detailed info about all containers

Being able to view detailed info for all containers on the cluster can make identifying problematic services and resolving performance problems much easier.

You can easily get a list of all running containers using docker node:

docker node ps $(docker node ls -q) | grep Running

However, if you want detailed information about CPU, memory, network and IO usage you can get that by combining docker node, cut, xargs and ssh then invoking docker stats on each node:

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER /usr/local/bin/docker stats --no-stream"
Output of running Docker Stats on all nodes on a cluster

Hopefully this has been helpful. If you have any tips for good Docker management tools would be great if you could share them below!

Iain Collins

Written by

News and media, civic tech and software. Cat herder at The Economist. Director at Glitch Digital.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade