NewSQL evaluation: TiDB and YugabyteDB (part 1)

ZaloPay is a payment app with the biggest user base in Vietnam. Handling large-scale transactions require us to heavily optimize computing infrastructure, esp. distributed databases. In these posts, we explain our evaluation for two major NewSQL databases including TiDB & YugabyteDB.

Motivation

Currently, there are many comparisons between TiDB and YugabyteDB but most of them are coming from exact companies that make the two databases. That leads to inclined judgments which are just self-praises.

Each database has its own advantages and drawbacks. Therefore, it can be only suitable for a certain problem and cannot cover all organization’s demands.

Methodology

When evaluating a database, not only the full power of that database should be exploited but its weaknesses also need to be analyzed deeply. That will show a clear trade-off when coming into deployment and real usages. All the above reasons require the ZaloPay teams to conduct a thorough benchmark between TiDB and YugabyteDB in an unbiased way.

In order to benchmark effectively, we have followed these milestones:

1. Diving: we researched each database to know how they work under the hood and which deployment model is most suitable to them.
2. Tuning: Common database-benchmark tools (Sysbench, YCSB) are used. We modify configurations & commands to be adaptable to our internal infrastructure.
3. Planning: we defined tests containing which operations are performed in the database, how to initialize data and way to control the number and ratio of operations during the benchmarks. Tests are categorized & executed via scripts.
4. Running: we ran benchmarks and analyzed the outputs.

For further details about a test plan and benchmark results, visit this repository.

In case you don’t have time to read, here are our findings in brief:

• For transactional operations that have a combination of read and write, TiDB has better performance than YugabyteDB’s.
• For operations that not use transactions, YugabyteDB has better throughput and latency than TiDB’s.

1. Deployment model

1.1 TiDB

We use TiDB version 3.0.12. The model includes:

• 3 nodes: 3 separate (TiDBs and Sysbench), 3 separate (TiKVs and PDs).
• 1 HAProxy

Hardware configuration

Details about version, configurations can be seen here.

1.2 YugabyteDB

We use YugabyteDB version 2.1.6.0. The model includes:

• 3 nodes: one YB-Server and one YB-Master are in the same node.
• 1 HAProxy

Hardware configuration

Details about version, configurations can be seen here.

2. Benchmark and evaluate

In our benchmark, we use two popular tools: Sysbench and YCSB (Yahoo! Cloud Service Benchmark). In this part, we report test results using Sysbench version 1.0.20.

2.1 Test configuration

We prepared workload data including 1 table and initialization of 30,000,000 records. There are 10 tests run, each test runs for 30 minutes. We also test concurrency by increasing the number of threads from 8 → 16 → 32 → 64 → 128. The following are comparisons and reviews of the benchmark results.

2.2 Test results

a. oltp_read_only with range selects

This test does only READ operations with transaction including select point, read in range (read in random range length, the sum of range, order the range, read distinct in range).

Evaluate:

• The test includes operations related to range such as select, sum, order, distinct.
• TiDB: We see that TiDB completed the test with spectacular results. Throughput increased slightly when we increased the number of threads and it reached a peak at 47.85 TPS (128 threads). However, P99 of TiDB in this test also increased from 434.83 ms (8 threads) to 4517.9 ms (128 threads).
• YugabyteDB: YugabyteDB seemed very poor when executing range queries. Latency P99 of Yugabyte reached the max value of the benchmark tool (100000ms). The highest throughput was just about 0.2 TPS.

In general, for queries related to reading in a range, TiDB outperforms YugabyteDB.

b. oltp_read_only without range selects

This test does only point select operations with transaction.

Evaluate:

• We re-tested the oltp_read_only but removed operations related to the range query. The result changed considerably.
• TiDB: throughput increased significantly when we increased the number of threads: from 466.29 TPS (8 threads) to 5934.76 TPS (128 threads). There was a downward trend in Latency P99 from 23.95ms to 21.11ms while throughput climbed at stage 8 threads to 32 threads. However, P99 increased again from 23.95ms (64 threads) to 31.95ms (128 threads)
• YugabyteDB: throughput increased from 238.82 TPS (8 threads) to 2301.47 TPS (64 threads) and suddenly grew 3 times to 6443.03 QPS when threads reached 128 threads. Although latency P99 decreased from 42.61ms (8 threads) to 38.25ms (64 threads), it turned upward afterward to 40.82ms (128 threads).

In general, there is the same upward trend in TiDB and YugabyteDB when gradually increasing the number of threads and the same lowest P99 latency at 32 threads: 21.11ms and 35.59ms respectively. Gathered results from each test case, it shows that TiDB has better throughput and latency than YugabyteDB.

c. oltp_write_only

This test does write operations with transaction like updating on an indexed field, updating on non-indexed field, delete then insert one field.

Evaluate:

• This test includes operations: update index/non-index, delete, inserts.
• TiDB: Latency P99 of TiDB was stable in range 32 threads → 128 threads although throughput increased through each stage. Throughput doubled but latency P99 rose 5 times when increasing threads from 8–16.
• YugabyteDB: As the number of threads increased so did throughput. In range 16 -> 32 threads, throughput doubled but latency P99 kept stable at 24.84ms. In range 64 -> 128 threads, throughput increased but latency P99 decreased suddenly from 53.85ms to 37.23ms.
• Looking at P99 latency results, YugabyteDB had lower P99 latency but it had lower throughput than TiDB. Let’s look through the average latency measurement below:

• Obviously, the latency Avg of TiDB was lower so it had higher throughput.

Based on the above results, we see YugabyteDB performs write operations than TiDB. Latency P99 of YugabyteDB is 2→5 times lower than TiDB’s. However, TiDB has higher throughput — 2 times than YugabyteDB has in some cases (8 threads, 16 threads).

d. oltp_write_only with the only delete_insert

This test just performs delete then insert operations with transaction.

Evaluate:

• The test script is the same as the script of c. oltp_write_only but keeps only delete_insert operations.
• TiDB: Throughput increased steadily as the number of threads increased. Latency P99 rose more than 5 times at stage 8 → 16 threads: from 17.32ms to 87.56ms. In stage 32 → 128 threads, latency P99 just increased slightly.
• YugabyteDB: throughput also increased about 4 times at stage 8 → 16 threads: from 1073.55 TPS to 1803.83 TPS. About latency P99, there was a decrease from 31.94ms at 8 threads to 18.28ms at 32 threads. However, it increased slightly again when threads increased from 32 → 128 threads.
• In 8 → 32 threads, YugabyteDB had lower P99 latency but it had lower throughput than TiDB. To understand, let’s look through the average latency measurement below:

In this test, YugabyteDB has 3→5 times lower latency P99 than TiDB has. Additionally, YugabyteDB’s throughput is also higher than TiDB’s in 32→64 threads.

e. oltp_read_write

This test combines operations of oltp_read_only and oltp_write_only mentioned above. We can also configure whether to execute read in range operations or not.

Evaluate:

• Mix test read and test write.
• TiDB: throughput increased as the number of threads increased. Latency P99 decreased about 2 times from 2238.47ms at 8 threads to 1089.3ms at 32 threads. However, it increased dramatically — 3 times from 2449.36ms at 64 threads to 6462.45ms at 128 threads but throughput did not change much.
• YugabyteDB: very poor performance. Latency P99 reached a peak of 100000 ms.

In this test, there are some queries in a range which impacted YugabyteDB’s performance. Meanwhile, TiDB still completed this test with an acceptable result. We will consider to re-benchmark this test without range operations.

f. oltp_point_select

This test does only point select operations like oltp_read_only without range selects but without transaction.

Evaluate:

• TiDB: throughput doubled at stage 8 → 32 threads. Latency was still stable and got the lowest at about 5ms when the number of threads was 32 and increased slightly afterward.
• YugabyteDB: throughput also doubled at stage 8 → 32 threads. Latency P99 decreased from 7.56ms at 8 threads to 4.65ms at 64 threads. It got the lowest at 4.91ms when the number of threads was 32.

In general, two databases have approximate throughput. Regrading to latency P99, TiDB is more stable than YugabyteDB but they both have the lowest latency when benchmarking with 32 threads.

g. oltp_insert

This test does only insert operations without transaction.

Evaluate:

• TiDB: throughput increased as the number of threads increased. However, latency was stable which changed from 11.04ms to 15.14ms.
• YugabyteDB: throughput increased 3 times at stage 64 → 128 threads from 21869.33ms to 68535.28ms. Latency in that range also increased but not considerably. Latency P99 was quite low which fluctuated from 3.82ms → 8.51ms

In general, YugabyteDB outperforms TiDB in write both throughput and latency P99 but two databases are both stable while executing write operations.

h. oltp_update_index

This test only updates on an indexed filed without transaction.

Evaluate:

• TiDB: the same with oltp_insert, throughput increased when the number of threads increased but latency P99 was still stable.
• YugabyteDB: had good throughput which rose significantly from stage 32 → 128 threads. Latency P99 first increased slightly at stage 8 → 16 threads: from 11.04ms to 11.45ms. After that, there was a decrease in P99 at stage 16 → 128 threads: from 11.45ms to 5.83ms.

In general, YugabyteDB writes better than TiDB despite higher throughput.

i. oltp_update_non_index

This test only updates on a non-indexed field without transaction.

Evaluate:

In general, the result is the same as oltp_update_index’s. Currently, we conclude that when updating the index, the value of the key’s index and the value of row containing that index need to be changed.

j. oltp_delete

This test does only delete operations without transaction.

Evaluate:

• TiDB: throughput increased as an increase in the number of threads. It doubled at stage 32 → 128 threads. Latency P99 was still stable which decreased from 8.9ms (8 threads) to 8.58ms (32 threads).
• YugabyteDB: Like TiDB, throughput has the same upward trend with the number of threads. Latency P99 increased considerably from 6.55ms (8 threads) to 15.00ms (64 threads). However, latency suddenly decreased to 6.85ms when benchmarking with 128 threads.

In general, the throughputs of the two databases are quite equal. However, latency P99 of TiDB is more stable, majorly in stage 16–64 threads.

2.3 Test summary using Sysbench

Throughout test cases of the Sysbench tool to test oltp, we give an understandable conclusion that TiDB is better in cases of read operations or mixed read-write operations. Meanwhile, YugabyteDB has outstanding write performance. In future work, there will be additional select for update benchmark (used in change bank balance) to provide a more precise evaluation.

See more details of our TiDB and YugabyteDB benchmark results using Sysbench.

In the next section, we will evaluate TiDB and YugabyteDB using YCSB.