Benchmarking an 18 Terabyte YugaByte DB Cluster with High Density Data Nodes
For ever-growing data workloads such as time series metrics and IoT sensor events, running a highly dense database cluster where each node stores terabytes of data makes perfect sense from a cost efficiency standpoint. If we are spinning up new data nodes only to get more storage-per-node, then there is a significant wastage of expensive compute resources. However, running multi-terabyte data nodes with Apache Cassandra as well as other Cassandra-compatible databases (such as DataStax Enterprise) is not an option. As DataStax Enterprise documentation notes:
To avoid problems, DataStax recommends keeping data per node near or below 1 TB. Exceeding this value has the following effects:
- Extremely long times (days) for bootstrapping new nodes.
- Impacts maintenance (day-to-day operations), such as recovering, adding, and replacing nodes.
- Reduces efficiency when running repairs.
- Significantly extends the time it takes to expand datacenters.
- Substantially increases compactions per node.
YugaByte DB is also a Cassandra-compatible database but is powered by a radically different replication and storage architecture. Replication is strongly consistent and is based on the Raft Consensus protocol. Storage engine, known as DocDB, is a Log Structured Merge (LSM) engine derived from Facebook’s RocksDB project. As highlighted in “How Does the Raft Consensus-Based Replication Protocol Work in YugaByte DB?”, bootstrapping of new nodes and removal of existing nodes are much simpler and more resilient operations when compared to the eventually consistent Cassandra-compatible databases.
Additionally, there are no background read repair and anti-entropy operations to worry about. This is because the write path is always strongly consistent based on synchronous replication. There is nothing to repair at read time (or in the background) and reads can be served correctly without using any quorum. These architectural differences mean that YugaByte DB can theoretically support much higher data density per node while remaining high throughput and low latency. Question is how much higher? This post highlights that YugaByte DB is now able to run a 18 TB cluster of 4 x 4.5 TB data nodes with high reliability and performance while easily handling day 2 operational tasks such as adding/removing nodes.
Originally published at blog.yugabyte.com.
