MongoDB: Creating a 3-Node Replica Set Cluster

By the end of this article, you will be able to deploy your own replica set cluster consisting of three separate Mongod processes.

Let’s start with A quick introduction to Replica Sets:

A replica set in MongoDB is a group of mongod processes that maintain the same data set. Replica sets provide redundancy and high availability, and are the basis for all production deployments.
These Mongod processes usually run on different nodes(machines) which together form a Replica set cluster.

Put simply, this means that all the data that is written on any particular node of this cluster is replicated across all the nodes. This provides redundancy to the data, thereby ensuring high availability and durability of the data even if a node goes down (temporarily or even permanently).

Let’s see how we can deploy a MongoDB Replica set consisting of 3 nodes. Practically a 3-node cluster will consist of 3 separate Mongod processes running on 3 different nodes(or machines). So in case any of the nodes fail, the remaining nodes will participate in an election and elect a new PRIMARY node by a process called `Failover`. This ensures data availability and durability.

Here in our case, just for the sake of simplicity, we will deploy all the three Mongod process on a single node(machine), so that we can have a simulation of a 3-node cluster running locally, where each Mongod instance will be running in isolation and have its own dbpath ( the directory where Mongod instance saves all database related data).

The three Mongod processes will run on ports 27011, 27012 and 27013 as specified in the respective Mongod conf files.

Note: Mongod is MongoDB’s daemon process.

Let’s begin.

Part 1: Creating a keyfile. We need to create a keyfile that will be common for all the nodes. This keyfile is used for intra-cluster authentication between the nodes.

Note: If the mongod processes were actually running on different machines, then each machine will have a copy of this keyfile.

To create the keyfile and set appropriate permissions on it, issue the following commands. Here we are using OpenSSL for creating the keyfile.

sudo mkdir -p /usr/local/var/mongodb/pki/
openssl rand -base64 741 > /usr/local/var/mongodb/pki/replicaset-keyfile
chmod 400 /usr/local/var/mongodb/pki/replicaset-keyfile

Part 2: In this step, we will create Mongod config files to start Mongod processes.

Below is the Mongod config file for the first Mongod process (to simulate the first node). Save it as node1.conf

storage:
  dbPath: /usr/local/var/mongodb/node1
net:
  bindIp: localhost
  port: 27011
security:
  authorization: enabled
  keyFile: /usr/local/var/mongodb/pki/replicaset-keyfile
systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/node1/mongod.log
  logAppend: true
processManagement:
  fork: true
replication:
  replSetName: replicaset-example

Create the required directories for db path and log path.

mkdir -p /usr/local/var/mongodb/node1
mkdir -p /usr/local/var/log/mongodb/node1

Now we do the same thing for the second Mongod process (to simulate the second node).

Below is the Mongod config file for the second node. Save it as node2.conf in the same directory as node1.conf

storage:
  dbPath: /usr/local/var/mongodb/node2
net:
  bindIp: localhost
  port: 27012
security:
  authorization: enabled
  keyFile: /usr/local/var/mongodb/pki/replicaset-keyfile
systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/node2/mongod.log
  logAppend: true
processManagement:
  fork: true
replication:
  replSetName: replicaset-example

Again create the required directories for dbpath and logpath using the following command.

mkdir -p /usr/local/var/mongodb/node2
mkdir -p /usr/local/var/log/mongodb/node2

Finally, below is the Mongod config file for the third Mongod process (to simulate the third node). Save it as node3.conf in the same place as node1.conf and node2.conf

storage:   
  dbPath: /usr/local/var/mongodb/node3
net:   
  bindIp: localhost  
  port: 27013
security:   
  authorization: enabled   
  keyFile: /usr/local/var/mongodb/pki/replicaset-keyfile
systemLog:   
  destination: file   
  path: /usr/local/var/log/mongodb/node3/mongod.log  
  logAppend: true
processManagement:   
  fork: true
replication:   
  replSetName: replicaset-example

And for the last time, we need to create dbpath and logpath directories for this third node as well. This is because we want each Mongod instance to run in isolation.

mkdir -p /usr/local/var/mongodb/node3
mkdir -p /usr/local/var/log/mongodb/node3

Now, once all the config files are ready and the directories for dbpath and logpath are created, we are all set to launch our Mongod processes. Navigate to the directory where you saved the above three config files and execute the following commands.

mongod -f node1.conf
mongod -f node2.conf
mongod -f node3.conf

If you want, you can verify that the Mongod processes are indeed running. Feel free to check this in any way that you want. One way to do this is by using the following command.

ps aux | grep -v grep | grep mongod

The above command will list all the Mongod instances that are running. You should see three instances.

Woohoo!! This step was a long one. But now all our Mongod processes are running. Currently, they are running fairly independently and in isolation and are still not a part of the same replica set. In the next final step, we will create a replica set and add these three Mongod processes to it.

Part 3: Connecting to and adding nodes in our Replica Set.

• Connect to any of the nodes using the mongo shell. We’ll connect to first node.

mongo --port 27011

• Once logged in to the mongo shell, execute the following commands in the shell.

rs.initiate(); // to initiate the replica set
use admin;     //switch to admin database for creating root user
db.createUser({
 user: "test-user",
 pwd: "test-pass", 
 roles: [{ role: "root", db: "admin"}]
});

• Now disconnect from the shell using CTRL+C or exit command and login to the replica set using the above credentials for the new root user. We had to create the above root user because it’s necessary to authenticate, to carry out replica set related operations.

mongo --host "replicaset-example/localhost:27011" -u "test-user" -p "test-pass" --authenticationDatabase "admin"

• Once logged in, try giving the following command:

rs.isMaster()

If you look at the output of the above command, you see that in the “hosts” field it only lists a single host. This indicates that currently, we have only one node in the replica set.

To add other nodes (Mongod processes) do the following:

rs.add("localhost:27012") //to add second node running on port 27012
rs.add("localhost:27013") //to add third node running on port 27013

Now try running rs.isMaster() again. You will see the following output.

// Output of rs.isMaster(). Only relevant fields shown here. 
{
...
  "hosts" : [
   "localhost:27011",
   "localhost:27012",
   "localhost:27013"
  ],
  "setName" : "replicaset-example",
  "secondary" : false,
  "primary" : "localhost:27011",
  "me" : "localhost:27011",
  "ok" : 1,
...
}

Meaning of the above output is as follows:

1. hosts: This field lists the hosts(nodes) in the replica set. We can see that there are three nodes listed here. That means we have three nodes in this replica set.

2. setName: This field shows the name of the replica set. This was configured in the Mongod config file of the three nodes.

3. me: This field gives the name of the current node to which mongo shell is connected to.

4. secondary: This boolean field being false in this case shows that the current node is not a SECONDARY node.

5. primary: This field indicates the PRIMARY node of the replica set. In this case it is localhost:27011

Yayyy!! Our Replica Set is up and running.

Now whenever we connect to the replica set, the mongo shell will automatically connect to the PRIMARY node. This is because the nodes have an internal way of communicating with each other and every node will redirect the mongo shell connection to the PRIMARY.

Any write operation that we do now, will be replicated across all the nodes for redundancy and availability purposes.

BONUS!!

Whenever one of the nodes in a Replica Set goes down, there is an internal election by which all the nodes (which have the right to vote) participate in an election (literally!) and elect a new PRIMARY among themselves. This is called Failover.

We can simulate this node failover. To observe this behavior we will make the PRIMARY node step-down using the following command.

rs.stepDown();

Once we execute the above command in the mongo shell, we will notice a slight delay before the output. This delay is because of the internal election taking place to choose a new PRIMARY.

Now try executing rs.isMaster() once again. Now if we observe the output, we can see that a different node has been elected as the PRIMARY.

Now that we have reached the end of this article, we were able to create and deploy our own replica set. Feel free to write any data on any node and see it replicate on others as well.

In production, all the nodes of this replica set will be different machines each running their own Mongod process.

Thanks for reading this article. Follow me for more informative articles.