MongoDB Speed Secrets: The Critical Role of Disk IOPS and Throughput.
Introduction:
In the dynamic world of database management, performance bottlenecks can arise from unexpected sources. Recently, we faced a perplexing issue: as our application scaled, we observed a surprising increase in MongoDB response times. After a thorough investigation, we pinpointed the culprit: low disk throughput.
In this blog, I will share our real-world experience with this issue. We’ll explore how an increase in IOPS and Disk Latency negatively impacted our application’s performance, delve into the technical reasons behind this phenomenon, and outline the steps we took to identify low disk throughput as the root cause. By understanding the intricate relationship between disk throughput and database performance, you can better anticipate and mitigate similar challenges in your own systems.
What are Disk IOPs?
IOPS (Input/Output Operations Per Second) is a performance measurement used to quantify the number of read and write operations that a storage device (like a hard disk drive (HDD), solid-state drive (SSD), or network storage) can perform in a second. IOPS is a crucial metric in evaluating the performance of storage devices, especially in environments with high data processing demands, such as databases and large-scale applications.
Mongo Operations vs DISK IOPs(read more)
- find: 1 IOP
- insert: 1 IOP + Number of indexes
- delete: 1 IOP + Number of indexes
- update: 2 IOPs + Number of indexes
Note: In real time, a single find query in MongoDB can indeed result in multiple IOPS, particularly when dealing with large result sets. Each batch retrieval beyond the initial set involves additional I/O operations, and the efficiency of these operations is influenced by factors like disk throughput and indexing.
How did changes to our underlying storage volume type lead to longer response times at database layer?
We decided to migrate our volume type to the newer one because of its cost advantage. However, following the switch, we noticed a spike in response times for listing queries. These queries, which involved find, reads, and writes, were notably affected. Initially, our focus was on evaluating client-specific changes, application configuration updates, and query page sizes to identify any modifications from integrations. Despite dedicating significant time to tracking down these potential causes, we found no conclusive evidence. With a hint from the APM indicating unusual getMore calls from the application to the database, we shifted our focus to infrastructure metrics(volume-level metrics). Surprisingly, we observed increased disk latency for both read and write operations, prompting us to shift our investigation towards evaluating disk metrics.
How did we understand the increase in Disk Latency and its correlation with query times?
Upon reviewing the IOPS graphs, we delved into the MongoDB logs to analyze potential query explains that could provide further insights. An intriguing pattern emerged in find queries, revealing a notable increase in subsequent queries known as “getMore” queries. In situations of poor disk IOPS Smaller batch sizes are often favored to reduce the data fetched from disk per batch, thereby easing strain on the storage subsystem. However, limitations in disk IOPS may hinder the server’s ability to efficiently handle larger cursor batch sizes, potentially leading to increased query response times. In such cases, a reduction in cursor batch size due to poor disk throughput may prompt the client to quickly exhaust the initial batch of documents, necessitating more “getMore” calls to fulfill query requirements.
While additional “getMore” calls aid in retrieving necessary data, they introduce extra overhead in terms of network latency and server processing. MongoDB strives to strike a balance between minimizing “getMore” calls and mitigating strain on the storage subsystem caused by poor disk throughput.
What are getMore calls?
When a query is executed, MongoDB attempts to retrieve as many documents as possible in the first batch(defaulting to 101 documents per batch). If all requested documents are not fetched in the initial query, a cursor is opened, and subsequent fetches are made using getMore calls to retrieve the remaining documents.
Why do Poor disk throughput = Increased getMore calls?
When disk access is slow, the data fetch might be slower, resulting in smaller batches being retrieved. Smaller batches mean more getMore calls are needed to complete the data retrieval. Each getMore call introduces additional latency and load, leading to an overall increase in query response time.
What is the relationship between disk throughput and IOPS?
The relationship between disk throughput and IOPS is crucial in the context of retrieving data from MongoDB, as it directly impacts the performance and efficiency of database operations.
If the database operations are small (e.g., reading individual documents), the performance is more dependent on IOPS. For example, a high number of small reads/writes will require high IOPS to maintain performance.
For large read/write operations (e.g., retrieving a large document or scanning a collection), disk throughput becomes more critical. Higher throughput allows MongoDB to move more data in and out of the disk quickly.
Let’s consider an example to illustrate this relationship:
Suppose a MongoDB deployment on an SSD has a maximum throughput of 500 MB/s and can handle 50,000 IOPS. If MongoDB operations involve reading 4 KB documents, the throughput can be calculated as:
Throughput (MB/s) = IOPSĂ—Size of each I/O operation (MB)
Throughput = 50,000 IOPSĂ—4KB = 50,000 Ă— 4/1024MB/s = 195.31MB/s
In this scenario, while the maximum throughput of the SSD is 500 MB/s, the effective throughput based on the IOPS and the size of the I/O operations is 195.31 MB/s. This demonstrates how the size of the operations and the IOPS limit can constrain the effective throughput.
How does mongodb manage data?
The WiredTiger storage engine is the default storage engine for MongoDB, designed to provide high performance, scalability, and efficiency. Its support for document-level locking, compression, MVCC, and efficient memory usage makes it suitable for a wide range of applications, from high-concurrency environments to storage-heavy and write-intensive workloads. Proper configuration and tuning of WiredTiger can further optimize MongoDB’s performance to meet specific application needs.The WiredTiger storage engine in MongoDB uses two primary types of data structures to manage data: B-trees and LSM (Log-Structured Merge) trees.
B-trees are balanced tree data structures that maintain sorted data and allow searches, sequential access, insertions, and deletions in logarithmic time. Each node in a B-tree can have multiple children, and the tree is kept balanced by ensuring that all leaf nodes are at the same depth.
Inserting a document in a B-tree involves locating the appropriate node and modifying it. This can cause multiple I/O operations, especially if nodes need to be split to maintain the B-tree properties.
Searching for a document in a B-tree involves traversing the tree from the root to the leaf nodes. Each traversal step may require reading a node from disk.
LSM(Log-Structured Merge) trees are designed to handle high write throughput by batching write operations in memory (in a structure called a memtable) and periodically flushing them to disk (into structures called SSTables). The LSM tree periodically merges and compacts these SSTables to maintain efficiency.
Inserts are initially written to an in-memory structure (memtable), which is fast. Periodically, the memtable is flushed to disk as an SSTable.
Searching in an LSM tree involves checking the memtable and then searching through multiple SSTables on disk.
LSM trees periodically merge and compact SSTables to maintain efficiency.
How does IOPS impact the performance with these data structures?
- Slower insertions, updates, and deletes in B-trees as each node operation waits for disk I/O.
- Delayed searches in B-trees due to slow node retrieval.
- Slow flushing and compactions in LSM trees, causing potential blocking during writes and slower searches due to multiple SSTable reads.
Overall, high IOPS enhances the performance of both B-tree and LSM tree structures by reducing the latency of disk operations, while low IOPS can severely degrade their performance by introducing significant delays in read and write operations.
What was the oversight here and how did we resolve?
As we delved into the relationship between IOPS and throughput, we discovered that our new volume had a default limited throughput of 128 MBps and 3000 IOPS. In contrast, the old volume offered a throughput of 250 MBps with the same 3000 IOPS(existing). Despite having sufficient IOPS, the lower throughput on new volumes prevented optimal scaling of IOPS. By increasing the throughput, we achieved better utilization of IOPS, which significantly enhanced MongoDB performance.
Conclusion
Our analysis revealed the critical impact of IOPS on MongoDB query times and overall performance. Poor disk throughput leads to more “getMore” queries, increasing query response times due to additional overhead. High IOPS reduces latency in disk operations, enhancing data structure performance, while low IOPS introduces delays.
The relationship between disk throughput and IOPS is also crucial. Small operations depend on IOPS, while large operations rely on throughput. Optimizing disk IOPS and throughput is essential for efficient MongoDB operations, ensuring fast query responses and robust system performance.
