MongoDB Load & Crash Testing Process
Hello, in this post, I will talk about the load and crash testing processes we performed for MongoDB POC.
First, let’s briefly mention MongoDB and the roles of MongoDB nodes. MongoDB is an open-source NoSQL database. Each record is a document and stored in JSON format.
Mongos Node: The server where applications requests will be met and fulfilled.
Config Node: The server where the shard and replica configurations are stored.
Shard Node: The database server where the data is stored.
Let’s start by mentioning the environments we used for the load testing process and its results.
Load Testing Scenario
Our purpose in the test we conducted was to find the database limits and bottlenecks in MongoDB nodes.
The server nodes we used for this purpose were as follows;
- 3 mongos nodes
- 3 config nodes
- 2 shard nodes.
Our load testing scenario was an intense writing process.
While carrying out the load testing scenario, we observed two distinct CPU behaviors on the nodes, and these were 4-core and 8-core CPUs. 8GB memory was employed on servers with a 4-core CPU, whereas 16GB memory was employed on servers with an 8-core CPU.
- Test Clusterı
2. Test Clusterı
MongoDB version 4.4.2 (Nov 18, 2020) was preferred.
Since MongoDB supports transactions, all transactions we made ran according to transaction ACID (atomicity, consistency, independence, durability). And they were written as commits.
We worked on 10 million record inserts each, with the same load we used on the nodes with different CPU resources. Since the CPU difference is a factor that affects the insert performance, we observed significant differences between 4 and 8-core CPUs, and this was reflected in the metrics.
Let’s look through the metrics of the test.
- Runtime
We saw that the time spent on the CPU and the total time decreased far more at the 8-core CPU, and actually, the Throughput accounted for this.
- Throughput
While the 4-core CPU ran approximately 10 thousand transactions per second, this value increased to 13–14 thousand at 8-core CPU. Even we managed to raise this value up to 20 thousand with the configurations.
- MinLatency
- MaxLatency
We saw that the 4-core CPU reached 23 times higher maximum latency than the 8-core CPU.
- Avg Latency
We saw that average latency reduced to one-tenth when the CPU doubled.
In the load testing we performed, we made an observation based on the CPU difference, but to make the most of MongoDB’s performance;
- Making the right directories
- Database design
- Application design
- Database configuration
- Enough available memory
Since MongoDB stores most of the manageable data in memory, one of its strengths is query performance.
Crash Testing Scenario
In the test we conducted, our purpose was to observe the up/down statuses of our cluster by trying out different scenarios for each component on MongoDB. The server nodes we used for this purpose were as follows;
- 3 mongos nodes
- 3 config nodes
- 2 shard nodes.
In the crash testing, we performed 10 scenarios on Replica and Primary nodes.
- Shard Replica Node Down
In the first scenario, when a shard replica node is down, an error is received if there is a structure that statically goes to that replica node, and here, we can see the importance of SDK configuration. Apart from that, the shard status, cluster status, and business impact will be in the up case.
2. Shard Primary Node Down
When a shard primary node is down, this will produce an effect equal to the failover time. Following the failover process, it will proceed to the up case.
3. Config Node Down
When a config node is down, the shard status, cluster status, and business impact will be in the up case since our config cluster does not have a single node.
4. Mongos Node Down
When a mongos node is down, the shard status, cluster status, and business impact will be in the up case since our mongos cluster does not have a single node.
5. Config Node Down
When two config nodes are down, the shard status, cluster status, and business impact will be in the up case since our config cluster does not have two nodes.
6. Mongos Node Down
When two mongos nodes are down, the shard status, cluster status, and business impact will be in the up case since our config cluster does not have two nodes.
7. Config Node Down
When three config nodes are down, the shard status, cluster status, and business impact will be down since all config nodes will be down, and hence, support is required.
8. Mongos Node Down
When three mongos nodes are down, the shard status, cluster status, and business impact will be down since all mongos nodes will be down, and hence, support is required.
9. Shard Replica Node Down
When two shard replica nodes are down, the shard status, cluster status, and business impact will be down since there is no more up shard node, and hence, support is required.
10. Shard Primary Node Down
When two shard primary nodes are down, the shard status, cluster status, and business impact will be down since there is no more up shard node, and hence, support is required.
In the database architecture, the architectural structure can be executed by calculating all possible cases that can remain up as far as possible, but the extent of the usability of the structure varies by the architecture of the physical servers, virtualization architecture, DB architecture installed, and its configuration, and client status.
We have completed our test scenarios for MongoDB. I hope you find it useful.
Sincere regards.
