Program test analysis and technical advice on EOSIO
Abstract
Huobi Research conducted a research and analysis of the Dawn 3.0 version of EOSIO through evaluating its platform performance in multiple scenarios.
Based on our test conditions (AWS servers in LAN environment), EOSIO can reach 1900 TPS at its best performance. There is a discrepancy between our result and the average TPS (3000), optimal TPS (6000), and theoretically optimal TPS (8000) result in the Dawn 3.0 documentation, but the worst performance of 1000 TPS described in the document can be stably reached.
Along with increasing server configuration, the maximum TPS of the system will also increase. At the same time, if the conditions allow, increase server resources of the super node (Block Producer) obtains better overall performance. Therefore, in the main network environment, we recommend using servers with the highest possible configuration.
Decentralized deployment can result in an adverse effect on performance. Since the current EOSIO program code does not support multi-threading, only a single -core of CPUs can be utilized, so parallel computing cannot be used to improve efficiency. In this situation, decentralized deployments, especially adding super-nodes, do not enhance performance for alleviating single server pressure . Instead, they increase CPU overhead and may have a negative impact on TPS.
Reasonable use of JIT will enhance performance. This method can reduce the CPU usage and increase TPS to a certain extent, but its impact on stability is also noticeable.
1. Foreword
Currently EOS super-node campaign is rolling full stream. According to statistics, as of May 2, there are 102 active bidding organizations . At the same time, block.one released EOSIO’s Dawn 3.0 version in April and plans to go live on main network in June.
Earlier, numbers of communities have set up test networks and tested EOSIO performances, especially on number of transactions per second (TPS). However, TPS performance is subject to many conditions such as compiler parameters, server hardware, operating systems, and networks. The test results revealed more of the “theoretical fuel consumption” under laboratory conditions (reference: https://steemit.com/cn/@eoseoul/bmt-eosio-tps-by-eoseoul-block-one-jit).
Unlike many of the previous tests, the research and analysis conducted by us Huobi Research does not focus on whether EOSIO can achieve the 1M TPS declared by Dan Larimer’s team. Instead, we analyze our observations of the performance indicators of Dawn 3.0 under different conditions to propose ideas and suggestions for future main network architecture and server configuration. Huobi Research will publish follow up study if new versions such as Dawn 4.0 is released.
It should also be noted that the indicator data obtained from this particular test is not and should not be regarded as proof or confirmation of the final performance result of the EOSIO platform or project. Hereby declare.
2. Main Conclusions
After analyzing our test results, we obtained and proposed the following main conclusions and technical advices:
l Based on our test conditions (AWS servers in LAN environment), EOSIO can reach 1900 TPS at its best performance. There is a discrepancy between our result and the average TPS (3000), optimal TPS (6000), and theoretically optimal TPS (8000) result in the Dawn 3.0 documentation (Reference: https://medium.com/eosio/eosio-dawn-3-0-now-available-49a3b99242d7), but worst scenario 1000 TPS described in the document can be stably reached.
Along with increasing server configuration, the maximum TPS of the system will also increase. At the same time, if the conditions allow, increase server resources of the super-node (Block Producer) obtains better overall performance. Therefore, in the main network environment, we recommend using servers with the highest possible configuration.
Decentralized deployment can result in an adverse effect on performance. Since the current EOSIO program code does not support multi-threading, only a single-core of CPUs can be utilized, so parallel computing cannot be used to improve efficiency. In this situation, decentralized deployments, especially adding super-nodes, do not enhance performance for alleviating single server pressure . Instead, they increase CPU overhead and may have a negative impact on TPS.
Reasonable use of JIT will enhance performance. This method can reduce the CPU usage and increase TPS to a certain extent, but its impact on stability is also noticeable.
3. Descriptions on test conditions
3.1 Testing Program Version
The testing program is version dawn-v3.0.0 of EOSIO
3.2 Hardware Environment
Amazon AWS. Listed below according to the level of configuration：
AWS EC2 t2.large (hereinafter referred to as the low-profile server): 2-core 2.3 GHz, Intel Broadwell E5–2686v4 CPU, 8GB RAM
AWS EC2 C5.4xlarge (hereinafter referred to as high-profile server): 16-core 3GHz, Intel Xeon Platinum 8124M CPU, 32GB RAM
We used 10Gbps LAN between servers, communication delay (ping) is less than 1ms.
The operating system is Ubuntu 16.04.
3.3 Testing Methods and Tools
We used the txn_test_gen test plug-in tool provided by block.one to send test transaction data and observe performance indicators such as actual TPS and system CPU usage (Reference: https://github.com/EOSIO/eos/wiki/Testnet-Single-Host-Multinode; https://gist.github.com/spoonincode/fca5658326837b76fd744d39b2a25b4e).
4. Test Scenarios and Results Analysis
4.1 Scenario 1
Deploy a "super node" (Block Producer, hereinafter referred to as BP) and a normal node (Full Node, hereinafter referred to FN) on the same low-profile server.
The results achieved by sending test data through FN are as follows:
4.1a
The results achieved by sending test data through BP are as follows:
4.1b
It is worth noting that when the transmission rate is 1700, BP’s CPU has been used 100%, but BP still shows stable processing capacity. At the same time, the usage rate of the CPU where the FN’s processing is located fluctuates greatly and cannot be synchronized from BP to block in time. With the system running, the block height difference between BP and FN will increase.
Through the results of this scenario, we come to the following conclusion:
Because the current EOSIO program does not support multi-threading, it is not possible to use the parallel computing of multi-core CPUs to improve efficiency.
BP's CPU usage is lower than FN's CPU usage. This finding seems to be the contrary of the usual understanding of "super nodes." We believe that the reason lies in two nodes undertaking different works: BP's design philosophy is to ensure transaction security and integrity (not losing blocks), and FN is designed to provide services externally, requiring transaction signatures, serialization , verification, etc.;
When the deals are transmitted through BP and FN respectively, there are significant differences in the platform TPS, which is also related to the above-mentioned tasks such as the signature and serialization of transactions.
4.2 Scenario 2
Improve server configuration and continue to observe system performance. Deploy a Block Producer (hereinafter referred to as BP) and a Full Node (hereinafter referred to as FN) on the same high-profile server.
The results achieved by sending test data through FN are as follows:
4.2a
The results achieved by sending test data through BP are as follows:
4.2b
Comparing with Scenario 1, we observed that the maximum TPS of the system increases when the server configuration increases.
4.3 Scenario 3
4.3a
The results achieved by sending test data through BP are as follows:
4.3b
Comparing with Scenario 2, we observed that the CPU usage in this scenario is slightly higher than that of the same condition, and this is believed to be caused by the cost of processing network traffic. Separate deployment in this scenario does not affect TPS significantly. This also relates to the current single-threaded approach: when BP and FN are deployed on the same server , they do not compete for CPU resources. Therefore, if the current code does not support multi-threading, it is possible that separate deployment will not enhance performance, but instead increase the CPU overhead.
4.4 Scenario 4
In Scenario 3, BP and FN are deployed on the same configuration server. Scenarios 4 and 5 deploy BP and FN on different servers with different configurations. In this scenario, BP is deployed on a low-profile server and FN is deployed on a high-profile server.
The results achieved by sending test data through FN are as follows:
4.4a
The results achieved by sending test data through BP are as follows:
4.4b
4.5 Scenario 5
In this scenario, FN is deployed on a low-profile server and BP is deployed on a high-profile server.
The results achieved by sending test data through FN are as follows:
4.5a
The results achieved by sending test data through BP are as follows:
4.5b
Comparing Scenario 4 and Scenario 5, we believe that allocating more resources for BP creates better overall performance.
4.6 Scenario 6
The above scenarios utilized two servers (1 BP, 1 FN). In this scenario we add in one FN to deploy one BP and two FNs on three high-profile servers.
The results achieved by sending test data through FN are as follows:
4.6a
The results achieved by sending test data through BP are as follows:
Comparing the results with that of Scenario 2, we conclude that adding a server in a single-threaded scenario increases CPU overhead and may negatively affect TPS.
4.7 Scenario 7
The above scenarios are single BP cases. We increase the nodes to 2 BPs and 2 FNs in this scenario
The results achieved by sending test data through FN are as follows:
4.7a
The results achieved by sending test data through BP are as follows:
4.7b
Comparing with the results in Scenario 6, it can be seen that the CPU usage after adding one more BP will increase, and the maximum TPS achieved will also be significantly reduced, indicating that increasing BP has a greater impact on performance. If multiple nodes are distributed in off-site main network environments, complex environmental factors such as network delays will intensify the impact. Thus, higher requirements on node server resources is imposed.
4.8 Scenario 8
Dawn 3.0 description file mentioned that the default interpreter of this version was changed from JIT to Binaryen WebAssembly, which will weaken performance but increase stability and reduce the delay when compiling smart contracts. In Scenario 8, we change the operating parameters of scenario 2 to JIT and make observation.
The results achieved by sending test data through FN are as follows:
4.8a
The results achieved by sending test data through BP are as follows:
4.8b
Comparing to Scenario 2, we see that CPU usage is significantly reduced. However, test interruption occurs when the CPU is not fully used, which proves that JIT does affect system stability.
