Architecting a Robust MongoDB Cluster
Setting up a single-instance MongoDB cluster on a local machine may seem straightforward, but when it comes to a high-throughput, highly available production environment, a simplistic MongoDB design falls short.
Consider a scenario where numerous users interact with your application, generating thousands of read-and-write operations on your on-premise database. Scaling up your single-node instance or migrating to a cloud-based Platform as a Service (PaaS) solution may be apparent choices, especially considering the reduced administrative overhead in terms of setup, replication, and scaling. However, not all businesses can afford the associated costs, or perhaps your company manages sensitive data that must remain within your corporate boundaries.
Whether you decide to operate on an on-premise server or opt for a cloud-based Infrastructure as a Service (IaaS) platform like AWS EC2, achieving an in-house implementation of MongoDB is a commendable pursuit.
Before delving further, let’s introduce a fundamental concept of MongoDB: the “replica set.”
A replica set in MongoDB is a group of mongod processes that provide redundancy and high availability. The members of a replica set are primary and secondary nodes.
In a MongoDB replica set, each member is assigned a voting weight, which dictates their role in selecting a new primary node.
A “voter” is a member that participates in the election process with a vote count of 1. This means it has a say in the election of the primary node. In contrast, you can also have non-voting members in a replica set. Non-voting members do not participate in elections and are often used for read-only purposes or to provide additional data redundancy without influencing primary node elections.
In summary, a MongoDB voter is a member of a replica set with a vote count of 1, and it plays a crucial role in the election of the primary node to maintain high availability and data redundancy.
Let’s address some common questions:
Q: What is the maximum number of members in a replica set?
A: Replica sets can have up to 50 members.
Q: What is the maximum number of voting members?
A: Replica sets can have up to 7 members permitted to vote.
Q: Why is an odd number (3, 5, 7) of voters recommended?
A: Using an odd number of nodes ensures that, in the event of a network partition, commonly referred to as a split-brain scenario, a clear majority can be established within one group of nodes. This majority is vital for the selection of a new primary node. In contrast, an even number of nodes may result in an inability to form a majority, hindering the primary node selection process.
A reliable architecture that works for us
Our recommended architecture consists of a six-node replica set, comprising one primary, four secondaries, and a delayed node. All but the delayed node can vote and are available for servicing client requests (hidden = false).
The delayed member retains copies of the replica set’s data set but reflects an earlier, delayed state. It serves as a “rolling backup” or historical snapshot, aiding in recovery from various types of human errors. Due to its purpose, it is kept hidden.
The delayed member cannot become the primary (hidden = true and priority = 0). It maintains a copy of the primary’s data set but remains invisible to client applications. In our use case, it caters to workloads with distinct usage patterns compared to the other replica set members.
Final thoughts
In a world that’s increasingly centered around cloud-based solutions, there are situations where creating and overseeing our own database clusters becomes essential. Companies must carefully assess the pros and cons of scaling out and maintaining multiple nodes of a database. In the end, being prepared for split-brain scenarios not only safeguards valuable time and revenues but also lightens the load on engineers, even if it entails additional redundancy costs.
