Deploying a MongoDB replica set on AWS Docker Swarm

To learn more about MongoDB I want to set up a replica set on a Docker Swarm within AWS. My knowledge about the database is very limited, so if you’re also a MongoDB beginner this walk-through could be helpful. The only requirement is an AWS account and basic knowledge about Linux server administration.

Creating the Docker Swarm

The article “Docker for AWS setup & prerequisites” in the Docker documentation is a great way to start. Click on the button “Deploy Docker Community Edition (CE) for AWS (stable)” and you will be redirected to AWS.

A Docker Swarm consists of two types of nodes: managers and workers. Managers maintain the cluster state and workers are instances whose sole purpose is to execute containers. You can read more about them in the article “How nodes work”. A replica set needs at least 3 MongoDBs. Why don’t we use 1 manager and 3 workers for that? By default a manager can execute containers and manage the Swarm at the same time. That’s why a typical testing environment for Docker Swarm consists of 3 managers and 0 workers.

You should have an SSH key ready or create one. I sometimes forget to clean up my Docker Swarm regularly with docker system prune, so I enabled the automatic daily resource cleanup. I can also recommend to enable CloudWatch for container logging and creating EFS prerequisites and I will tell you later why exactly.

Choose the instance type t2.micro and use the default values for everything else. The last step is the creation of the stack. Hopefully you will see CREATE_COMPLETE in your CloudFormation area after a few minutes.

There should be 3 instances in the overview and they should be in 3 different availability zones. Choose one of them and connect using SSH, the public IP and your key. There is one leader, but it doesn’t matter where you execute the following commands.

Deploy the MongoDB services

Save the following code in a file called docker-compose.yml and upload it to the server you connected. You can use SCP or SFTP for the file transfer.

If you deploy this YAML with docker stack deploy -c docker-compose.yml m it will create 3 services on your swarm and 3 volumes where the data will be saved. The stack is called m and all services and volumes will have that prefix. MongoDB by default only binds to localhost and the option --bind_ip_all is an easy way out, but not a very secure one. Also note that you usually want to have some form of authentication which is done using --auth, but we will omit that for our testing environment.

The volumes are 1 GB of SSD storage and are connected to the services with the amazing Cloudstor driver provided by Docker. If one host is not available the container will automatically move to a new, healthy host. That’s nothing fancy on a regular Docker Swarm, but the Cloudstor driver can attach the SSD storage to the new host as well. Three key factors are responsible for the high reliability of this configuration:

  • MongoDB — If the primary MongoDB service isn’t reachable, a new one will be elected. The fail-over process generally completes within a minute.
  • Docker Swarm—If a service fails, there will be an automatic restart every 30 seconds. If a node in the Swarm isn’t reachable, the services and SSD storage will move to a new node. This process shouldn’t take more than 3 minutes.
  • AWS — The nodes are automatically spread over the availability zones, in this example eu-central-1a, eu-central-1b, eu-central-1c. The nodes get tagged with the label availability_zone which we use to spread the services as well. The line - spread: node.labels.availability_zone in the YAML does that.

You can check the status of your deployment with the command docker stack ps m . The volumes are EBS storage and it can take a while before everything is running. If anything doesn’t work you would usually use for example docker service logs m_mongodb1 to find errors. In this case however we enabled CloudWatch for container logging and the Docker Swarm logs will be empty. There will be a new CloudWatch log group called Docker-lg, if you named your Swarm Docker during the creation.

Initiate the replica set

So far we created a Docker Swarm on AWS and have 3 MongoDB services running on it. They are not a replica set yet, because it needs to be initiated. If you are still connected to a node, then we can continue with the configuration. We need to know the container ID of the MongoDB services on the host we are connected to. The ID is in the first column when you execute docker container ls. Look for the container with the image mongo:3.6.3 and copy the ID. Then execute docker exec -it 7c827b57beef /bin/bash if your ID is 7c827b57beef.

Open the shell with mongo. Initiate the replica set with the following query. mongodb1, mongodb2 and mongodb3 are the names of the services in this case. You can learn more about this process in the article “Deploy a Replica Set”.

The answer should look like this:

You can see additional information about your replica set with rs.status(). Your shell should now start with either rs0:SECONDARY> or rs0:PRIMARY>. The MongoDB is part of a replica set named rs0 in the named role. You are probably on the primary MongoDB, because that’s where the initiation started. If you are not, you can see who the primary is with rs.status(). Do you want to try inserting and querying data? Have a look at the MongoDB documentation:

In the default configuration you may not read data from secondaries. You can change this behavior if you execute rs.slaveOk() on a secondary. There’s a lot to learn about replication and we didn’t take care off security or advanced configuration in this example. I have a few ideas in mind what to do next: