Updated for release v1.12.3: minor command changes once again, labels now applied at the node level, and no more need for custom mongo image.
Running a MongoDB Replica Set on Docker 1.12 Swarm Mode: Step by Step
With swarm mode, Docker 1.12 introduces some exciting new features for orchestrating a highly available cluster of services. This video from DockerCon 2016 gives a great overview, and there are many other excellent sessions online as well.
Docker swarm mode services are a collection of tasks (containers for now) dynamically allocated across the cluster. This is good for many reasons, but it means your application’s containers may move from one box to another, leaving their data volumes behind. For a database, that could be a sub-optimal situation. Fortunately many database systems already have high-availability baked in. All we have to do is keep the containers in one place, and the DBMS will take care of the rest.
Strategy
The basic plan is to define each member of the replica set as a separate service, and use constraints to prevent swarm orchestration moving them away from their data volumes. This preserves all the operational benefits that Docker provides, while nullifying the redundant (and in this case harmful) fault recovery features.
A brief note on conventions used in this post. I will bold everything that I typed into the terminal. I apologize for the verbosity and line wrapping of the output, but I think it is important to show everything that is happening. Also, don’t attempt to copy/paste any commands from the non-fixed-width paragraphs; Medium insists on “fixing” text, borking hyphens, quotes, spacing, and other important punctuation.
Inspect the Environment
For this example I will go from scratch with a Docker Toolbox 1.12.3 installation and set up a highly available Node.js app with a MongoDB 3.2 replica set as the data store. I will be doing this on my Windows 7 laptop, but the commands should be identical on other platforms. First thing to do is open the Docker Quickstart Terminal and get the lay of the land.
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.comStart interactive shellBlake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Running tcp://192.168.99.100:2376 v1.12.3Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 23:26:11 2016
OS/Arch: windows/amd64Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 23:26:11 2016
OS/Arch: linux/amd64
Provision Docker Machines
I don’t need the default machine, so I’m going to get rid of it, and create three new ones for the cluster. As this is a development environment, I will only have one cluster manager node, and two worker nodes. In production multiple managers should be used for fault tolerance.
Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine rm -f default
About to remove default
Successfully removed defaultBlake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine create --driver virtualbox manager1
Running pre-create checks...
Creating machine...
(manager1) Copying C:\Users\Blake.Mitchell\.docker\machine\cache\boot2docker.iso to C:\Users\Blake.Mitchell\.docker\machine\machines\manager1\boot2docker.iso...(manager1) Creating VirtualBox VM...
(manager1) Creating SSH key...
(manager1) Starting the VM...
(manager1) Check network to re-create if needed...
(manager1) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env manager1Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine create --driver virtualbox worker1
Running pre-create checks...
Creating machine...
(worker1) Copying C:\Users\Blake.Mitchell\.docker\machine\cache\boot2docker.iso to C:\Users\Blake.Mitchell\.docker\machine\machines\worker1\boot2docker.iso...
(worker1) Creating VirtualBox VM...
(worker1) Creating SSH key...
(worker1) Starting the VM...
(worker1) Check network to re-create if needed...
(worker1) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env worker1Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine create --driver virtualbox worker2
Running pre-create checks...
Creating machine...
(worker2) Copying C:\Users\Blake.Mitchell\.docker\machine\cache\boot2docker.iso to C:\Users\Blake.Mitchell\.docker\machine\machines\worker2\boot2docker.iso...
(worker2) Creating VirtualBox VM...
(worker2) Creating SSH key...
(worker2) Starting the VM...
(worker2) Check network to re-create if needed...
(worker2) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env worker2Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
manager1 * virtualbox Running tcp://192.168.99.100:2376 v1.12.3
worker1 - virtualbox Running tcp://192.168.99.101:2376 v1.12.3
worker2 - virtualbox Running tcp://192.168.99.102:2376 v1.12.3
PuTTY Interlude
Now that the VMs are running, I’m going to switch over from the Docker Quickstart Terminal to PuTTY. I find the interface much more user friendly. This is optional and doesn’t affect operation of the swarm cluster. I used PuTTYGen to convert each of the private keys to a format PuTTY understands. Then I connected to each VM via SSH with the username “docker” and the private key for authentication. Here are all the relevant keys, after I did the conversion.
Blake.Mitchell@BMITCHELL7X64 MINGW64 ~
$ find .docker/machine -type f -name id_rsa\*
.docker/machine/machines/manager1/id_rsa
.docker/machine/machines/manager1/id_rsa.ppk
.docker/machine/machines/manager1/id_rsa.pub
.docker/machine/machines/worker1/id_rsa
.docker/machine/machines/worker1/id_rsa.ppk
.docker/machine/machines/worker1/id_rsa.pub
.docker/machine/machines/worker2/id_rsa
.docker/machine/machines/worker2/id_rsa.ppk
.docker/machine/machines/worker2/id_rsa.pubCreate the Swarm
Next I’ll initialize the swarm on the manager, and join both of the workers. Most of this comes directly from the Getting started with swarm mode tutorial. It is important to specify the “ — listen-addr” parameter when using docker-machine and VirtualBox, as the VMs are assigned multiple network interfaces.
docker@manager1:~$ docker swarm init --listen-addr 192.168.99.100:2377 --advertise-addr 192.168.99.100:2377
Swarm initialized: current node (1w9qe9ceo8wfjyawytt4lu04m) is now a manager.To add a worker to this swarm, run the following command:docker swarm join \
--token SWMTKN-1-2b8xrkg7ymzmb9db8yyzglkqur8k32vfoeyd64d5dppv9d6ikl-f1k3vk7ohmrl9lzgvf4ah34tj \
192.168.99.100:2377To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions....docker@worker1:~$ docker swarm join --token SWMTKN-1-2b8xrkg7ymzmb9db8yyzglkqur8k32vfoeyd64d5dppv9d6ikl-f1k3vk7ohmrl9lzgvf4ah34tj --listen-addr 192.168.99.101:2377 --advertise-addr 192.168.99.101:2377 192.168.99.100:2377
This node joined a swarm as a worker....docker@worker2:~$ docker swarm join --token SWMTKN-1-2b8xrkg7ymzmb9db8yyzglkqur8k32vfoeyd64d5dppv9d6ikl-f1k3vk7ohmrl9lzgvf4ah34tj --listen-addr 192.168.99.102:2377 --advertise-addr 192.168.99.102:2377 192.168.99.100:2377
This node joined a swarm as a worker....docker@manager1:~$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host null overlay
Swarm: active
NodeID: 1w9qe9ceo8wfjyawytt4lu04m
Is Manager: true
ClusterID: c99d2hmp1sc2qhut99key1opt
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 192.168.99.100
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.27-boot2docker
Operating System: Boot2Docker 1.12.3 (TCL 7.2); HEAD : 7fc7575 - Thu Oct 27 17:23:17 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.8 MiB
Name: manager1
ID: 3HBM:NWLJ:WHID:GT7Z:ZCS4:HEX2:JATN:WGXH:K2JU:6ERB:U7HI:ULNP
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 33
Goroutines: 133
System Time: 2016-11-07T03:01:47.552799151Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
Insecure Registries:
127.0.0.0/8
You can see from the “docker info” output, I now have a three node swarm with one manager.
Preparing for MongoDB
It order to keep the mongo services pinned to the same nodes as their data volumes, I am setting a label “mongo.replica” on each of the nodes. I will use these labels in constraints when creating the services later.
docker@manager1:~$ docker node update --label-add mongo.replica=1 $(docker node ls -q -f name=manager1)
1w9qe9ceo8wfjyawytt4lu04m
docker@manager1:~$ docker node update --label-add mongo.replica=2 $(docker node ls -q -f name=worker1)
69ik3w7jzm2icbp1y9id0jaln
docker@manager1:~$ docker node update --label-add mongo.replica=3 $(docker node ls -q -f name=worker2)
eqlebvmdcofwvm827v7em4423The replica set will need an overlay network to allow the replicas to communicate, and accept connections from the application. Each replica also needs a volume for data and configuration. I will give each of the volumes a numerical suffix that matches the “mongo.replica” label for the underlying node. This is not required, but helps me keep track of them.
docker@manager1:~$ docker network create --driver overlay --internal mongo
brp2kd5xgul4vgyz5wtksles0
docker@manager1:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
87c4b1ec3724 bridge bridge local
68c3e2d453fe docker_gwbridge bridge local
df1270a391f5 host host local
96memqbs2j6j ingress overlay swarm
brp2kd5xgul4 mongo overlay swarm
fb1bc79742ed none null local
docker@manager1:~$ docker volume create --name mongodata1
mongodata1
docker@manager1:~$ docker volume create --name mongoconfig1
mongoconfig1
docker@manager1:~$ docker volume ls
DRIVER VOLUME NAME
local mongoconfig1
local mongodata1...docker@worker1:~$ docker volume create --name mongodata2
mongodata2
docker@worker1:~$ docker volume create --name mongoconfig2
mongoconfig2
docker@worker1:~$ docker volume ls
DRIVER VOLUME NAME
local mongoconfig2
local mongodata2...docker@worker2:~$ docker volume create --name mongodata3
mongodata3
docker@worker2:~$ docker volume create --name mongoconfig3
mongoconfig3
docker@worker2:~$ docker volume ls
DRIVER VOLUME NAME
local mongoconfig3
local mongodata3
Create the MongoDB Services
Now it’s time to create the MongoDB services that will comprise the replica set. I will use the official mongo image from DockerHub.
docker@manager1:~$ docker service create --replicas 1 --network mongo --mount type=volume,source=mongodata1,target=/data/db --mount type=volume,source=mongoconfig1,target=/data/configdb --constraint 'node.labels.mongo.replica == 1' --name mongo1 mongo:3.2 mongod --replSet example
b29ftmx77l75owmhmkswntcmz
docker@manager1:~$ docker service create --replicas 1 --network mongo --mount type=volume,source=mongodata2,target=/data/db --mount type=volume,source=mongoconfig2,target=/data/configdb --constraint 'node.labels.mongo.replica == 2' --name mongo2 mongo:3.2 mongod --replSet example
1o08g3f0b6ub60et2u7uu9bc5
docker@manager1:~$ docker service create --replicas 1 --network mongo --mount type=volume,source=mongodata3,target=/data/db --mount type=volume,source=mongoconfig3,target=/data/configdb --constraint 'node.labels.mongo.replica == 3' --name mongo3 mongo:3.2 mongod --replSet example
715k8yuv3uodxn2cyai43uxy2
docker@manager1:~$ docker service ls
1o08g3f0b6ub mongo2 1/1 mongo:3.2 mongod --replSet example
715k8yuv3uod mongo3 1/1 mongo:3.2 mongod --replSet example
b29ftmx77l75 mongo1 1/1 mongo:3.2 mongod --replSet example
docker@manager1:~$ docker service inspect --pretty mongo1
ID: b29ftmx77l75owmhmkswntcmz
Name: mongo1
Mode: Replicated
Replicas: 1
Placement:
Constraints : node.labels.mongo.replica == 1
UpdateConfig:
Parallelism: 1
On failure: pause
ContainerSpec:
Image: mongo:3.2
Args: mongod --replSet example
Mounts:
Target = /data/db
Source = mongodata1
ReadOnly = false
Type = volume
Target = /data/configdb
Source = mongoconfig1
ReadOnly = false
Type = volume
Resources:
Networks: brp2kd5xgul4vgyz5wtksles0The command to create each service is long, let me break the first one down a little: we’re asking docker to create a new service, with a single instance, on the “mongo” network, mounting the “mongodata1” and “mongoconfig1” volumes, running on the node where the label “mongo.replica” has value “1” with service name “mongo1”, using the image “mongo:3.2” and run the command “mongod -replSet example”.
It will take some time to pull the images the first time, use the “REPLICAS” column from “docker service ls” to see when all the service tasks are running. (Docker service replicas and MongoDB replicas are not the same thing, the former are redundant tasks (containers) in a service managed by the swarm, where as the latter are MongoDB instances participating in a replica set.)
Initiate the Replica Set
Next I’ll set up the MongoDB replica set. I am following the Deploy a Replica Set instructions from the documentation. First I will initiate the replica set with a single member configuration, then add the other two members. Because the services are on a private network, I will exec the mongo client already present in the running container to issue commands. This also demonstrates how to use one of the swarm assigned labels to identify containers.
docker@manager1:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7fafd0318920 mongo:3.2 "/entrypoint.sh mongo" 5 minutes ago Up 5 minutes mongo1.1.1hy6x18wn3x8qr2g827sjo66m
docker@manager1:~$ docker exec -it $(docker ps -qf label=com.docker.swarm.service.name=mongo1) mongo --eval 'rs.initiate({ _id: "example", members: [{ _id: 1, host: "mongo1:27017" }, { _id: 2, host: "mongo2:27017" }, { _id: 3, host: "mongo3:27017" }], settings: { getLastErrorDefaults: { w: "majority", wtimeout: 30000 }}})'
MongoDB shell version: 3.2.10
connecting to: test
{ "ok" : 1 }
docker@manager1:~$ docker exec -it $(docker ps -qf label=com.docker.swarm.service.name=mongo1) mongo --eval 'rs.status()'
MongoDB shell version: 3.2.10
connecting to: test
{
"set" : "example",
"date" : ISODate("2016-11-07T03:36:44.510Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 1,
"name" : "mongo1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 777,
"optime" : {
"ts" : Timestamp(1478489792, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-11-07T03:36:32Z"),
"infoMessage" : "could not find member to sync from",
"electionTime" : Timestamp(1478489792, 1),
"electionDate" : ISODate("2016-11-07T03:36:32Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 2,
"name" : "mongo2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 22,
"optime" : {
"ts" : Timestamp(1478489792, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-11-07T03:36:32Z"),
"lastHeartbeat" : ISODate("2016-11-07T03:36:44.448Z"),
"lastHeartbeatRecv" : ISODate("2016-11-07T03:36:43.872Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "mongo1:27017",
"configVersion" : 1
},
{
"_id" : 3,
"name" : "mongo3:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 22,
"optime" : {
"ts" : Timestamp(1478489792, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-11-07T03:36:32Z"),
"lastHeartbeat" : ISODate("2016-11-07T03:36:44.448Z"),
"lastHeartbeatRecv" : ISODate("2016-11-07T03:36:43.874Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "mongo1:27017",
"configVersion" : 1
}
],
"ok" : 1
}
docker@manager1:~$ docker exec -it $(docker ps -qf label=com.docker.swarm.service.name=mongo1) mongo --eval 'rs.config()'
MongoDB shell version: 3.2.10
connecting to: test
{
"_id" : "example",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 1,
"host" : "mongo1:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "mongo2:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "mongo3:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {},
"getLastErrorDefaults" : {
"w" : "majority",
"wtimeout" : 30000
},
"replicaSetId" : ObjectId("581ff6b54865b4277baf414d")
}
}
There it is, a healthy replica set!
Put an App on it
Now that I have a functioning replica set, I’m going to put a Node.js app in front of it to demonstrate the capabilities. I’m going to use the post-it board Koalab, because it gives a nice visual and everything is stored in MongoDB. The publisher of Koalab does not push it to Docker Hub, so I have taken care of that.
docker@manager1:~$ docker service create --env 'MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/koalab?replicaSet=example' --name koalab --network mongo --replicas 2 --publish 8080:8080 kalahari/koalab
44r0hriq9k5n314gulo7etmcb
docker@manager1:~$ docker service ps koalab
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
bykl6954h3veukal8tuw57rt7 koalab.1 kalahari/koalab worker2 Running Preparing 21 seconds ago
7jrfb3aibdp1kp1zntxp2gmk6 koalab.2 kalahari/koalab manager1 Running Preparing 21 seconds ago
docker@manager1:~$ docker service inspect --pretty koalab
ID: 44r0hriq9k5n314gulo7etmcb
Name: koalab
Mode: Replicated
Replicas: 2
Placement:
UpdateConfig:
Parallelism: 1
On failure: pause
ContainerSpec:
Image: kalahari/koalab
Env: MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/koalab?replicaSet=example
Resources:
Networks: brp2kd5xgul4vgyz5wtksles0
Ports:
Protocol = tcp
TargetPort = 8080
PublishedPort = 8080As you can see, my MONGO_URL addresses the entire replica set, and I attached this service to the “mongo” network. Now I can hit port 8080 on any of my swarm hosts and create a new post-it board. Even though there are only two copies of kalahari/koalab running, the magic of swarm networking makes the service available everywhere.
Thanks for taking the time to read, I hope it was informative, and allows you to benefit from my trial and error. If you have any questions, feel free to ask below, or ping me on twitter.
