NewSQL evaluation: TiDB and YugabyteDB (part 2)
In part 1, we used Sysbench to evaluate TiDB and YugabyteDB. This time, YCSB is used instead to test the serving capability of a database with the same deployment model in the previous benchmark.
1. Test configuration
The initial dataset has one table with 30,000,000 records. There were 5 workloads used for the benchmark, each test runs for 20 minutes. Similar to the benchmark using Sysbench, we also tested concurrent processing by increasing the thread count.
2. Test result
a. Workload A: Heavy workload for update operation (50% read, 50% update)
Evaluate:
- TiDB: throughput increased 2 times at stage 64 → 128 threads. In this threads range, P99 also increased considerably from 6.95ms to 14.4ms (Read) and from 13.17ms to 19.23ms (Update). In the range 8 → 64 threads, P99 was quite stable.
- YugabyteDB: throughput increased gradually as an increase of threads but latency P99 decreased gradually in the range 8 → 32 threads. Nevertheless, it increased 7 times in the range 64 → 128 threads.
In this test, YugabyteDB quite has better throughput and P99 than TiDB has. However, for the test with 128 threads, YugabyteDB’s latency P99 is about 2 times of TiDB’s. TiDB had good latency P99 as there is an increase in throughput.
b. Workload B: Read mostly workload (95% read, 5% update)
Evaluate:
- TiDB: throughput increased at stage 8 → 32 threads but latency declined from 6.15ms to 4.98ms (Read) and from 11.46ms to 9.57ms (Update). At stage 32 → 64 threads, throughput nearly doubled but P99 did not increase much.
- YugabyteDB: throughput doubled at stage 16 → 32 threads but latency P99 decreased slightly from 4.65ms to 4.59ms (Read) and from 5.41ms to 5.11ms (Update). When the number of threads reached 128 threads, YugabyteDB had the highest throughput with latency P99 decreased from 6.05ms to 4.97ms (Read) and 6.62ms to 5.58ms (Update).
In general, YugabyteDB has slightly better throughput than TiDB has in both Read and Update. Read latencies P99 of two databases are approximate but about Update, YugabyteDB performs better than TiDB does.
c. Workload C: Read-only (100% read)
Evaluate:
- TiDB: latency declined in the range 8 → 32 threads: from 5.64ms to 5.07ms though there was a gradual increase of throughput: from 2284 QPS to 10092.61 QPS. After that, throughput doubled at stage 32 → 64 threads but latency P99 just changed from 5.07 to 6.8ms. At stage 64 → 128 threads, P99 increased significantly and reached a peak of 9.825ms at 128 threads.
- YugabyteDB: latency P99 was low but not stable. It had the best result at tests with 16 threads (3.82ms) and 64 threads (3.8ms). In the range 64 → 128 threads, throughput and latency P99 also nearly doubled: from 3.8ms to 6.77ms of latency P99.
In general, YugabyteDB has better throughput and latency P99 than TiDB has but they are not as stable as TiDB’s.
d. Workload D: Read the latest workload (95% read, 5% insert)
Evaluate:
- TiDB: throughput nearly doubled at stage 16 → 32 threads with a slight decrease of Read latency P99: from 5.64ms to 5.29ms. However, Insert latency P99 increased from 8.98ms to 9.32ms.
- YugabyteDB: had low latency P99 in both Read and Insert. About Read, the highest throughput was 20597.66 QPS with latency P99 of 4.04ms. Regarding Insert, max throughput was 1250.54 QPS with latency P99 of 4.9ms.
In general, the throughput of YugabyteDB is higher than TiDB. Remarkably, Insert latency P99 of YugabyteDB is 2 times lower than TiDB.
e. Workload F: Read-modify-write (50% read, 50% read-modify-write)
Evaluate:
- TiDB: Read latency P99 was quite stable in range 8 → 64 threads which changed from 5.05ms to 6.9ms. About RWM, latency P99 decreased in range 16 → 64 threads: from 15.67ms to 12.61ms with throughput increased from 1188.91 QPS to 4371.35 QPS.
- YugabyteDB: had high throughput and low latency P99 at stage 8 → 64 threads. Nevertheless, in the range 64 → 128 threads, Read throughput increased dramatically from 18303.46 QPS to 50000 QPS and Read latency P99 also increased 5 times: from 4.64ms to 20.44ms. Regarding RMW results, throughput increased from 9150.24 QPS to 24996 QPS and latency P99 increased 5 times from 8.79ms to 40.15ms.
In short, the majority of YugabteDB’s latency P99 is 2 times lower than TiDB’s. However, at the test with 128 threads, YugabyteDB has very high throughput but bad latency P99.
3. Test summary using YCSB
Throughout all the tests with YCSB, YugabyteDB is better than TiDB in most cases. However, the results of TiDB are not too bad, it has good throughput and its latency is quite stable when increasing threads through benchmark runs. We need to note that YCSB does not use transactions in benchmark tests. Therefore, it can be affirmed that for queries without transactions, YugabyteDB is superior to TiDB.
See more details of our TiDB and YugabyteDB benchmark results using YCSB.
4. Issues
During the tests, we encountered some issues. They contain already present problems:
- Multiple homogeneous errors: [Already present in (yb/consensus/retryable_requests.cc:239): see here
- Workload D and Duplicate Key Errors: see here
And problems relating to our benchmark process and internal infrastructure:
- Workload D: The workload needs to be passed with the value of recordcount to know how many records are currently in the database. Therefore, it can generate an id that is not duplicate within the corresponding range of recordcount. However, we perform benchmark on 2 servers (2 * 64 threads), each server has the same recordcount and it must run the workload in parallel. Therefore, there is a problem with duplicate-key insert. Currently, each time the workload D finished for each case, we must re-insert the data to ensure the record is always consistent and avoid duplicate-key inserts when running the workload.
- Workload E: YugabyteDB has poor performance in read-range operations. That problem results in an extremely long and unbreakable workload even though the benchmark time exceeded maxexecutiontime value.
5. Summary
- Due to the long benchmark time that was nearly 35 hours if running continuous tests, we can not perform many times for each test to give the best result.
- Using 2 tools Sysbench and YCSB cannot tell the strength of both databases, but we can make some judgments about the 2 databases to apply them to the actual problems.
- According to the benchmark statistics above, we see that TiDB is capable of performing Read, Read-Update transactions like the select, select update, select delete very well, whereas YugabyteDB performs well in write transactions like insert, delete, update.
- For queries that do not use transactions, both TiDB and YugabyteDB have shown their abilities, but YugabyteDB is better than TiDB in throughput and latency P99.
6. Acknowledgment
Special thanks to my dear fellows, esp. Tai Pham, Thieu Vo, Dung Tran for instructing me about benchmarking databases with several tools. Moreover, they helped me to analyze & provide unbiased conclusions.
References
The following are some public benchmark results made by TiDB and YugabyteDB:
