Menapay Blockchain Tests: Quorum TPS
Payment systems and using cryptocurrency as an alternative to fiat currencies have been one of the most important use cases for the blockchain technologies since its start. Yet, there is still not a widely used blockchain based payment solution.
In Menapay, we are trying to build an enterprise level blockchain solution to replace traditional payment methods. Providing each and every good part of a standard process with an emerging technology, and finding better solutions to the parts it lacks are the challenges of this task. As Menapay development team, it is our job to follow the new technological solutions related to blockchains, analyze/test them to decide which works for us and modify/apply them regarding our interests. You can find a study of us about the comparison of permissioned blockchains here. In this post I will discuss some of the tests we have done on one of the best alternatives as the previous work suggests, which is Quorum.
Quorum is the enterprise focused version of Ethereum initiated by J.P. Morgan. The project aims to address challenges in financial industry by providing permissioned network, private transactions, high speed and high throughput. It supports smart contracts and it has an innate community by its Ethereum infrastructure. These features make it one of the best candidates for us to start working on.
The first aspect that we tested on Quorum was transaction speed. As comparable to its alternatives, Quorum is able to provide very high transaction speeds — hundreds of transactions per second (TPS). This is because of the Raft consensus algorithm it uses. Raft provides fast block times. (on the order of milliseconds instead of seconds) And fast block times generally means faster transactions. Quorum has tendency to create a new block immediately after a new transaction arrives, but it only creates one after if it’s been at least 50 ms since the last block. This is to prevent floods with creating immense amount of blocks.
To test TPS, we first created a setup with Quorum 7 nodes example. The example initializes 7 nodes with their own private transaction managers and the nodes all run on virtual machines. To apply a transaction load we used Chainhammer project. This project allows us to send transactions in different modes such as blocking/non-blocking; single threaded/multithreaded, sending public/private transactions, using web3/RPC or using Raft/IBFT as consensus. An example output from Chainhammer can be seen in Figure 1.
We performed several different experiments by changing the test variables. We chose Raft consensus as our standard during tests and never changed it. We always sent 1000 transactions. We also reset the chain and blocks for each test run. All transactions are sent to one specific node and read from another.
Peak values of avg. TPSes for these experiments are listed below:
Blocking, non-async, single-thread
Raft, web3, with public contract and txs — Peak: 115.8 TPS
Raft, web3 with private contract and public txs — Peak: 104.0 TPS
Raft, web3 with private contract and txs — Peak: 65.6 TPS
Raft, RPC with public contract and txs — Peak: 101.1 TPS
Raft, RPC with private contract and txs — Peak: 72.3 TPS
Non-blocking async multi-thread ( one thread per tx )
Raft, web3 with public contract and txs — Peak: 153.5 TPS
Raft, web3 with private contract and public txs — Peak: 143.6 TPS
Raft, web3 with private contract and txs — Peak: 78.6 TPS
Raft, RPC with public contract and txs — Peak: 110.2 TPS (cannot create 1000 connections and fails after ~500 txs.)
Raft, RPC with private contract and txs — Peak: 72.3 TPS (cannot create 1000 connections and fails after ~500 txs.)
Non-blocking async multi-thread queue with multi-workers
# of workers = 10
Raft, web3 with public contract and txs — Peak: 149.0 TPS
Raft, web3 with private contract and public txs — Peak: 148.2 TPS
Raft, web3 with private contract and txs — Peak: 86.0 TPS
Raft, RPC with public contract and txs — Peak: 139.7 TPS
Raft, RPC with private contract and txs — Peak: 81.2 TPS
# of workers = 20
Raft, web3 with public contract and txs — Peak: 159.2 TPS
Raft, web3 with private contract and public txs — Peak: 148.6 TPS
Raft, web3 with private contract and txs — Peak: 100.7 TPS
Raft, RPC with public contract and txs — Peak: 173.1 TPS
Raft, RPC with private contract and txs — Peak: 84.4 TPS
# of workers = 50
Raft, web3 with public contract and txs — Peak: 149.5 TPS
Raft, web3 with private contract and public txs — Peak: 147.3 TPS
Raft, web3 with private contract and txs — Peak: 89.5 TPS
Raft, RPC with public contract and txs — Peak: 128.6 TPS
Raft, RPC with private contract and txs — Peak: 78.1 TPS
# of workers = 100
Raft, web3 with public contract and txs — Peak: 151.0 TPS
Raft, web3 with private contract and public txs — Peak: 152.5 TPS
Raft, web3 with private contract and txs — Peak: 89.6 TPS
Raft, RPC with public contract and txs — Peak: 129.6 TPS
Raft, RPC with private contract and txs — Peak: 83.5 TPS
Non-blocking async multi-thread (one thread per tx) in batches of parallel threads
( batch size = 25 )
Raft, web3 with public contract and txs — Peak: 136.7 TPS
Raft, web3 with private contract and public txs — Peak: 130.1 TPS
Raft, web3 with private contract and txs — Peak: 83.7 TPS
Raft, RPC with public contract and txs — Peak: 130.7 TPS
Raft, RPC with private contract and txs — Peak: 81.1 TPS
RESULTS
- Comparing the protocols, Web3 performed slightly better than RPC for both public and private transactions in general. Web3 tests produced ~10–15 % more TPS than RPC.
- Public and private contracts with public transactions ended up with similar results. Even though public contracts gave better results than private contracts with public transactions, it might be wrong to generalize since it is not always and the amount is not so significant.
- Private transactions are 30–40% slower to process than public transactions. The operations performed by Constellation is most probably be the reason behind it.
- Multithread queue with multi workers seems the best method among other sending methods. Worker count is not directly proportional with the TPS. There seems to be a sweet spot around 20 workers where TPS is maximized. Then it drops again for larger amounts of workers.
After these trials, we also wanted to see how a high performance setup with real machines affect the test results. We created a setup with 3 high end computers. All three machines had 32 CPU cores and ~100GB RAM. We set up a node to each machine using quorum-maker. It is a tool to ease up the new Quorum node setup process. It also provides a nice dashboard to see system status, nodes, transactions, etc. We replicated the tests above to see how much we can increase the average TPS. The results were not so higher than the original one. After several trials we could only reach 175–180 TPS average at best case. The peak value seen was 206.1 TPS. During the traffic, max CPU used for the running geth process reached around 150–160%. But the overall CPU value was only around 2%. It can be concluded that the geth process cannot utilize multiple CPU cores. The memory used during the tests was also around 2% at its peak. So it is understandable why the number of CPUs or the amount of memory does not affect the test results that much.
The most critical thing during the tests was that the system lost responsiveness after a certain amount of time the peak TPS was reached. We tried it several times, and the nodes started not to process any newly incoming transactions after 5–10 minutes of continuous traffic. We tried to find similar problems on Quorum board, but we could not. If we decide to use Quorum after our tests, this problem should be one of our first priorities.
To conclude, Quorum provides faster block time and higher transaction speeds than its rivals. We tested Quorum for TPS to see whether it can satisfy our needs in Menapay. We will be testing continuously for other aspects than TPS, and other blockchain technologies than Quorum. Because, to find the best you should see the rest ;)
