Open in app

Sign in

Write

Sign in

MongoDB Cluster

Mongodb: Set-up sharded cluster for windows with just a double click.

MongoDB setting up Sharded cluster for windows with a double click.

Arun Pratap
Published in
7 min readApr 26, 2020

--

Please find Github repository to configuration file and script to set-up the environment at the end.

mongodb version 3.6+ , link to mongodb documentation wherever necessary for further reading.

Introduction to Sharded cluster.

Sharding is a method for distributing data across multiple machines. MongoDB uses sharding to support deployments with very large data sets and high throughput operations.

Database systems with large data sets or high throughput applications can challenge the capacity of a single server. For example, high query rates can exhaust the CPU capacity of the server. Working set sizes larger than the system’s RAM stress the I/O capacity of disk drives.

A MongoDB sharded cluster consists of the following components:

  • shard: Each shard contains a subset of the sharded data. Each shard can be deployed as a replica set.
  • mongos: The mongos acts as a query router, providing an interface between client applications and the sharded cluster.
  • config servers: Config servers store metadata and configuration settings for the cluster. As of MongoDB 3.4, config servers must be deployed as a replica set (CSRS).

Setting Up sharded cluster Environment in windows with 2-Shards having 3-Member Replica set Primary-Secondary-Secondary(P-S-S).

Mongodb sharded clustered set-up with 1- mongo router

We are going to set-up clustered env as in the diagram. (local set-up going with single router)

For the setup we will be spawning 9-mongod instances and 1-mongos(router) instance.

  • First we need to set-up config server replica Set.
  • Shard 01 replica set.
  • Shard 02 replica set.
  • Router(mongos) with the configuration pointing to Config Server’s.
  • Adding Shard 01 and Shard 02 to the Cluster through router.

For each instance will be creating a “db”(database files) and “log”(log files) folder.

Let’s Start……..

Setting up config 3-member replica set.

  • create folder structure for the 3-instances “db”, “log” folder
md csrs\csrs1\db csrs\csrs1\log csrs\csrs2\db csrs\csrs2\log csrs\csrs3\db csrs\csrs3\log
  • csrs1.conffile for one of the config server’s. We need to assign clusterRole: configsvr to mark it as a config server under common replSetName: csrs to make them member of the same replica set “csrs”.
sharding:
  clusterRole: configsvr
replication:
  replSetName: csrs
net:
  bindIpAll: true
  port: 26001
systemLog:
  destination: file
  path: csrs/csrs1/log/csrs1.log
  logAppend: true
storage:
  dbPath: csrs/csrs1/db/
  • Will do the same for rest of the 2-member’s of config replica set, by just changing the port if we running on the same host or can keep same if different host.
sharding:
  clusterRole: configsvr
replication:
  replSetName: csrs
net:
  bindIpAll: true
  port: 26002
systemLog:
  destination: file
  path: csrs/csrs2/log/csrs2.log
  logAppend: true
storage:
  dbPath: csrs/csrs2/db/

Save config to file:csrs2.conf

sharding:
  clusterRole: configsvr
replication:
  replSetName: csrs
net:
  bindIpAll: true
  port: 26003
systemLog:
  destination: file
  path: csrs/csrs3/log/csrs3.log
  logAppend: true
storage:
  dbPath: csrs/csrs3/db/

Save config to file:csrs3.conf

  • Now we can start the config server’s. running on port [26001, 26002, 26003]
echo starting Config replicaSet member's
 start mongod -f csrs1.conf
 start mongod -f csrs2.conf
 start mongod -f csrs3.conf
  • Config them to be part of same replica set.
// start one of the server client of the replica set.
 mongo --port 26001
 rs.initiate() 
// this will start showing it as primary.
// add 2 other config server to the replicaset.
 rs.add("didnasiina6:26002") 
// if ip doesn't work use machineName
 rs.add("didnasiina6:26003")
// check all replica set added.
 rs.conf() // this will print the info of the configured server.
{
        "_id" : "csrs",
        "version" : 3,
        "configsvr" : true,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "didnasiina6:26001",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {},
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "didnasiina6:26002",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {},
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "didnasiina6:26003",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {},
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {},
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5e09cfcc596228f43c03c4c2")
        }
}

Now we have config server replica set up and running.

Setting up Shard-01, 3-member replica set.

  • create folder structure for the 3-instances “db”, “log” folder
md sh01\sh011\db sh01\sh011\log sh01\sh012\db sh01\sh012\log sh01\sh013\db sh01\sh013\log
echo Created mongoDb folder structure for Shard 01 Server Instances
  • Setting up shard replicaset is exactly same what we already did with config replica set. will just have to tell this has a clusterRole: shardsvr and we are taking a different replSetName: sh01 that will be common across all the three member's of same shard replica set.
sharding:
  clusterRole: shardsvr
replication:
  replSetName: sh01
net:
  bindIpAll: true
  port: 27011
systemLog:
  destination: file
  path: sh01/sh011/log/sh011.log
  logAppend: true
storage:
  dbPath: sh01/sh011/db/

Save config to file: sh011.conf

will do the same for other 2-member sh012.conf and sh013.conf

  • Now we can start the shard server’s. running on port [27011, 27012, 27013]
echo starting Shard 01 Replica Set member's.
 start mongod -f sh011.conf
 start mongod -f sh012.conf
 start mongod -f sh013.conf
  • Config them to be part of same replica set.
// start one of the server client of the replica set.
mongo --port 27011rs.initiate()
// add 2 other config server to the replicaset.
rs.add("didnasiina6:27012")
rs.add("didnasiina6:27013")
rs.status() // this will print the info of the configured server.// ... not printing the output. already did for config replSet.

Now we have Shard 01 replica set up and running.

Setting up Shard-02, 3-member replica set.

It’s exactly the same as Shard 01 set-up. Just we have to do if find and replace sh01 to sh02 :) and assign new port’s for all the configuration and folder structure.

md sh02\sh021\db sh02\sh021\log sh02\sh022\db sh02\sh022\log sh02\sh023\db sh02\sh023\log
echo Created mongoDb folder structure for Shard 02 Server Instances
  • Server Configuration file sh021.conf, sh022.conf and sh023.conf
sharding:
  clusterRole: shardsvr
replication:
  replSetName: sh02
net:
  bindIpAll: true
  port: 27021
systemLog:
  destination: file
  path: sh02/sh021/log/sh021.log
  logAppend: true
storage:
  dbPath: sh02/sh021/db/
  • Now we can start the shard server’s. running on port [27021, 27022, 27023]
echo starting Shard 02 Replica Set member's.
 start mongod -f sh021.conf
 start mongod -f sh022.conf
 start mongod -f sh023.conf
  • Config them to be part of same replica set.
// start one of the server client of the replica set.
mongo --port 27021rs.initiate()
// add 2 other config server to the replicaset.
rs.add("didnasiina6:27022")
rs.add("didnasiina6:27023")
rs.status() // check for status, look for Primary/Secondary members.
rs.conf() // see the configuration of replSet.

Now we have Shard 02 replica set up and running.

Configuring Router.

  • Starting mongo router(mongos).

As mongo router is not a db instance with no database of it’s own will just need to create “log” folder.

md router\log
  • router config file mongos.conf we need router to point to the configDB: csrs/didnasiina6:26001,didnasiina6:26002,didnasiina6:26003 no storage section as it doesn’t have db of it’s own, uses config server to route the query to the shard’s.
sharding:
  configDB: csrs/didnasiina6:26001,didnasiina6:26002,didnasiina6:26003
net:
  bindIpAll: true
  port: 26000
systemLog:
  destination: file
  path: log/mongos.log
  logAppend: true
  • Add shard to monos(router).
// connect to mongos client 
   mongo --port 26000
   sh.addShard("sh01/didnasiina6:27011")
   sh.addShard("sh02/didnasiina6:27021")
// check the status
sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5e09cfcf596228f43c03c4d1")
  }
  shards:
        {  "_id" : "sh01",  "host" : "sh01/didnasiina6:27011,didnasiina6:27012",  "state" : 1 }
        {  "_id" : "sh02",  "host" : "sh02/didnasiina6:27021,didnasiina6:27022,didnasiina6:27023",  "state" : 1 }
  active mongoses:
        "4.0.10" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  5
        Last reported error:  Could not find host matching read preference { mode: "primary" } for set sh01
        Time of Reported error:  Tue Apr 21 2020 11:50:36 GMT+0530 (India Standard Time)
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                sh01    1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : sh01 Timestamp(1, 0)
        {  "_id" : "mars",  "primary" : "sh01",  "partitioned" : false,  "version" : {  "uuid" : UUID("039312c7-e98d-4099-b69a-1fc57de1d9e4"),  "lastMod" : 1 } }

All done.. Mongodb sharded Clustered Environment is UP and Running.

Some helpful commands.

Github Repository to configuration file and script to set-up the environment.

Please follow the README.md for quick effortless setup with just a double click :)

Github: https://github.com/arun2pratap/mongodbClusterForWindowsOneClick

This example if for local box will seamlessly work for distributed set-up. Just you have to make the folder structure create config file and start server on that host and when configuring need to point to that host.

Tried to keep it simple as possible, please add review/comment anything you see can be improved. Happy Codding :)

Hope this was helpful, Thank you

Mongodb
Sharding
Clustering
Mongodb Tutorial
Windows

--

--

Arun Pratap
Geek Culture

Written by Arun Pratap

71 Followers
Writer for

Programmer, Database enthusiast currently exploring/learning MongoDB.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams