Key considerations when configuring Private Ethereum Networks
Disclaimer: Opinions expressed are solely my own and do not express people, institutions or organizations that I may or may not be associated with in professional or personal capacity unless explicitly stated. Any views or opinions are not intended to malign any person, organization or company.
In this article I am going to cover the key points to take into consideration when setting an Ethereum Network. I am not referring how to install a node but instead the fundamental concepts that are not always clear. This article will be focused on Private Ethereum networks, in that way the network I will refer relies on the IBFT consensus protocol (instead of Ethash/PoW algorithms currently used in most public blockchains). Having said that, the following questions will be answered in this article:
- What is the amount of gas that needs to be set when creating a network?
- What are the resource requirements an Ethereum network needs?
- Does all nodes in the network requires the same resources or are those different?
- How different kinds of transactions affects the Ethereum network performance?
- What is the critical kind of transactions when setting a network, the lighter or the heavier ones?
- What is the transaction pool role when the network works at maximum capacity?
- What are the vulnerabilities that are addressed when setting a right amount gas for each block?
Preliminary concepts
Let’s start our analysis by explaining the role of the term “gas”. The gas is a measure of how much computation power (per node) is required to execute a transaction in an Ethereum network. Not all transactions are the same. In order to illustrate that I will show three examples:
- Adding a struct element into a mapping. For this simple example the gas cost is 213890 gas units. In the subsequent set of tests this kind of transactions will be referred as “type A”
- Execute a simple set in a smart contract, which gas cost is approximately 33314 units. In the subsequent set of tests this kind of transactions will be referred as “type B”
- Send ether from one account to another, which gas cost is 21004. In the subsequent set of tests this kind of transactions will be referred as “type C”
- Another key point to take into consideration is that all nodes in a network check the validity of an incoming block by executing all transactions and checking the cryptographic proofs associated with each transaction included in a block.
Now let’s take a look at the parameters set in the genesis file.
Note this is a little bit different for an IBFT network because it sets the following particular attributes:
- blockperiodseconds: States that each block is going to be produced each 2 seconds.
- gasLimit: This is the maximum amount of gas that each validator is willing to spend when validating a block, before adding that to the blockchain. In our case it is currently set to 0x2FEFD800 (in hexadecimal units) which is equivalent to 804247552 units in decimal format.
We can interpret both attributes with this sentence:
Each 2 seconds a new block is produced, with the maximum allowed amount of gas to spend of 804247552 units.
It means that literally we could support sending 804247552/21004 or 38290tx/block of ether transfer transactions, it is too much. Setting a big gasLimit in the genesis could be possible but is not feasible to achieve in ethereum networks.
According to my experience setting high values like that is not recommended for two main reasons, among other:
- If A transaction with high amount of gas is accepted in the network it could lead to later synchronization problems when new nodes pass through the synchronization process.
- Even when the first condition does not happen the synchronization process becomes slower when more transactions are validated on each block.
What we have to do is to set a more realistic “gasLimit” for the network. In the next section I am going to show a network for testing purposes in order to delimit this with the gasLimit per block.
Testing Network
Let’s sketch up the testing network
Simulating a real scenario, all transactions will be received from the regular nodes in the network. The two validator nodes are focused on the consensus algorithm in order to build the next block, they only communicate with the regular nodes to receive the transactions that will pass an agreement process between both validators before adding those to the next block.
Network Characteristics and considerations
- Consensus protocol: IBFT2
- Ethereum Client implementation: Hyperledger Besu
- Block time: 2 seconds
- All nodes use virtual CPUs, those are not the same as a physical one in the sense that a physical CPU core with hyper-threading is seen as two virtual CPU for the operating system. Find more info here.
The flow is the following:
- From the testing machine a configurable amount of transactions per second are sent to the network through the regular nodes (the type of transactions to send are also configurable)
- The regular nodes pass the received transactions to the validator nodes.
- The validator nodes adds the transactions to the next block.
- The validated block is broadcast to the regular nodes.
- The regular nodes checks the validity of the incoming block, for that they recompute all the root hashes in the block.
Set Test 1
The first set of tests uses the following characteristics for the nodes
For this test case, the genesis file was configured to have 800 Million gas (gas limit per block). Then I made many test cases varying the amount of transaction rate (transactions per second) for each test and also the testing time (from minutes to hours).
The first test made to the network is sending transactions of 213890 gas cost. After many tests with different rates, 32 tx/s (64 transactions per block)was a rate that passed 1 hour test.
Even when a rate of 32 tx/s is not too much, it is a limited amount because of the limited resources allocated to each machine. Here the machines that suffers more are the regular ones, because they have to process all the incoming requests (from the testing machine) whilst also receiving the blocks from the validator set.
From the previous test we could achieve 13688960 gas per block, considering now transactions of type B (33314 gas/tx) we could in theory send 205 tx/s of that type.
Instead of sending 205 tx/s I achieved a sustained response of 160 tx/s.
As shown in the image the response is sustained and also the real value of gas per block that was used was 10660480 units of gas, it is demonstrated in the following image:
Compared to transactions of type B (213890 gas/tx), more transactions of type A (33314 gas/tx) were successfully sent. Also different tests shows that trying fill to 13688960 gas on each block was not possible, instead only 10660480 units of gas was filled on each block. Why?
Filling 13688960 of gas on each block means 32 tx/s (type A) or 205 tx/s (type B). The test shows that whilst 32 tx/s is feasible, it is not true when sending 205 tx/s. That is because even when the amount of gas per block is the same for both cases, sending transactions of type B at a rate of 205 tx/s implies much more state tree calculations which requires more computational power; because of the limited resources of the testing network that rate is not possible, so the solution is reduce the rate of transactions that were sent to the network. As mentioned lines above the sustained response is achieved at 160 tx/s for transactions of type B. As a consequence the network is limited to 10660480 gas/block(instead of 13688960 gas/block), even when the genesis block have a much higher value (800 M).
Using the same reasoning as before, sending a bunch of transactions of type C (21004 gas/tx) will require much more power to process the transactions which in turn will limit to a value less than 10660480 gas/block.
It is important to remark that when transaction rates increase there is also a more required resources for nodes that exposes the RPC in order to receive the the incoming transactions from the client applications that send transactions to the network.
Another key point to take into consideration is what happens when transaction rates are higher than the ones a network can process (according to its resources). For example sending transactions of type C (21004 gas/tx) at a rate of 240 tx/s, more of them can be placed into a block. Because of the high gasLimit (800 M) is set in the the genesis file the validators try to add as many transactions as they can without considering their own resource limitation. The final effect is a heavy block produced with many transactions to be propagated to each regular node in the network. That heavy block implies more state changes to calculate to reach the final state root in the block, if the regular nodes have not enough CPU resources to process the incoming block and also receive all the incoming transactions through the RPC port then the node will fail.
Here is another example where 216 transactions per second (tx/s) of type B (33314 gas per transaction) were sent to the testing network, if all transactions were processed correctly each 2 seconds then 432 transactions could be placed on each block, but instead what is seen is:
Note those values are much higher than 432 transactions per block, it happens because of the explained above. This leads to issues on the node.
Let’s summarize this section with the following table:
The figure shows in green successful passed tests. The filled in red failed.
Note that higher rates are only achieved when the gas per transaction decreases (columns tx/s vs gas/s). Furthermore note how it is possible to send heavy transactions(more gas cost per transaction) at a lower rate but not possible to use the same amount of gas per block when trying to send lighter transactions at a higher rates, that is the case of the pair of items 1-2 and also the pair 4-5.
From this test two important conclusions can be extracted:
- First, not all transactions are the same, each set of transactions can generate a different behavior on the network. For a total amount of gas “g” , equivalent to a bunch of “n” transactions of type B (33314 gas/tx) those are computationally more expensive than “m” heavier transactions of type A (213890 gas/tx, like a method in a smart contract that modifies a struct). Where m<n.
- Second, too much gasLimit configured per block generates heavier blocks that could lead regular nodes without enough resources to fail; also increases the overload on validators because of the big amount of transactions to validate on each block.
Set Test 2
In order to not to extend too much this article I will show only the successful result.
The purpose of this second set of tests is to demonstrate that increasing the resources on the whole network allows to process much more transactions.
In this set of tests the resources were increased for the regular nodes (it was done in order to have more computing power to process more state changes in lighter transactions (transactions of type C) whilst also receiving new transactions to process through the RPC exposed port).
Also, in order to have a logical relation between the resources available on each node and the transactions that can be processed in the network I limited the maximum available amount of gas to a less value than 800 millions. For this set of tests the value was set to 16131072 (0xF62400) units of gas maximum allowed per block.
In the previous set of test I started the analysis with heavy transactions (type A) then I realized that transactions that limit the available gas to per block are the lighter ones (Like type C). So in this new set of tests the approach is to start testing transactions of type C and find a suitable transaction rate then find the correspondent tx/s values that can be achieved with the same amount of gas per block but with heavier transactions.
Due to the limitations in the stress test machines I used, the maximum amount of transactions (of type C) sent was 336 tx/s, it is notably a much more higher rate than transactions sent in the previous set of tests. As shown in the following image the response was excellent and a 100% of all sent transactions were successfully processed by the network.
For this test case, the validators are not impacted because they have enough resources.
Considering the successful amount of 336 tx/s, it is 14114688 gas/block we can infer that it is possible to send transactions of type A at a sustained rate 66 tx/block or ~32 tx/s. It is also possible to send transactions of type B at a rate of ~212 tx/s.
An important conclusion to get from this table is that we finally increased the resources in the network, to support a greater rate per second of lighter transactions (type B or C).
Note that the previous test sent 336 tx/s of type C, that is 672 tx/block or equivalent to 14114688 units of gas; that amount only represents the 87,5% of the total available gas per block or gas Limit (16131072).
What happens if client applications send a total equivalent amount of gas greater than the allowed in the genesis file?. In the previous section (set Test 1) it was clear that because the gas Limit per block was higher, validators added blocks with much more transactions than the network can really process; as a consequence the network started failing at higher rates of transactions. Now, because the gas limit has been set to a value that is in concordance with the network resources then validators start to queue transactions (in the transaction pool) that cannot be added to the block because of the gas Limit per block. Eventually those transactions will be added into a valid block or will be discarded if the transaction pool reaches its full capacity (in this scenario newer transactions have priority over old ones).
In a real scenario, in order to guarantee all transactions are processed, applications could delegate the management of transactions to queue managers like Kafka, there are great solutions out there ready to be used like ethconnect.
Let’s summarize this section:
- Increasing resources overall on the network takes special relevance with lighter transactions (transactions of type B or C). For example increasing the resources allowed to increase the rate of transactions of type B from 160 tx/s to 212 tx/s that is an increase of 32% .
- Setting an adequate amount of gas per block allows to protect the network and guarantee processing of all transactions without affecting the network performance.
- Additionally it could be useful to have some solutions (like ethconnect) that could help handle lots of transactions and make sure those will be processed according to what the network can process.
Answering questions
Based on the results obtained in the previous tests, let’s answer the questions formulated at the beginning of this article:
What are the resource requirements an Ethereum network needs?
Honestly the network can be as little as having 2 vCPUs and 8 GB-RAM for each machine, but you have to take into consideration that this setup establishes a limit in relation to what throughput you can achieve in the network.
What is the amount of gas that needs to be set when creating a network?
The maximum amount of gas that we are willing to spend for each block creation is strongly related to the resources assigned to validators. The more CPU assigned the more amount of “gasLimit” that can be set for each block.
Does all nodes in the network requires the same resources or are those different?
Some of the executed tests showed that regular nodes (the ones that exposes the RPC port) requires more power in order to properly receive all request while also checking and adding new validated blocks to its own blockchain database. In our test network we only had 4 regular nodes, but if more regular nodes were present then, on average, the rate of incoming transactions is attenuated.
How different kinds of transactions affects the Ethereum network performance?
Executed test also showed that it is possible to fill much more transactions of type C than type B. So we can reach higher levels of TPS (transactions per second or tx/s) like 336 tx/s of type C. While it is true it also means more computing power required for the whole system.
What are the critical kind of transactions to take into consideration when setting a network (the lighter or the heavier ones?)
In my view and according to the tests, for a total amount of gas “g” , equivalent to a bunch of “n” transactions of type C(21004 gas/tx) are computationally more expensive than “m” heavier transactions of type A (213890 gas/tx, like a method in a smart contract that modifies a struct). Where m<n.
So setting a network will really be influenced on how much of those transactions can be processed by the network. Like the tests, if we know that a network with 4 vCPU (2 physical CPU with hyper-threading) for each validator can support 336 tx/s of type C, then we can calculate how many transactions of another kind like transactions of type A or B can be supported. If it is not enough the next step is to increase the resources and test again with transactions of type C to find an appropriate value that fits your needs. It is worth to say that increasing the resources a lot doesn’t scale well because it also implies more barriers for new participants who want to join the network; so you should consider this criteria before increasing resources.
What is the transaction pool role when the network works at maximum capacity
Up to this point we know that a blockchain network cannot process all transactions at once, under high loads on the network some transactions are put on a memory space called the transaction pool, here transactions wait to be validated. The transaction pool plays an important role making sure that as long as transactions are stored here they will be added to a block later. Nevertheless there are other additional tools focused on managed queued transactions one of those solutions is ethconnect.
What are the vulnerabilities that are addressed when setting a right amount gas for each block?
Setting a gasLimit that is correctly related to the resources available in the network guarantees that all validated blocks will be correctly processed by all nodes in the network. Also it wont matter if at some point the validators are receiving many transactions to add to the next block, they will only add as much transactions as the gasLimit parameter allows them. All other transactions will remain in the transaction pool to be added in a future block.
Code
Interested to make some tests? Please find the available code here.
Conclusions
This article have covered some important points before building a private Ethereum Network. In general the throughput obtained in an Ethereum network will mainly depend on the resources it has and how this is aligned with the maximum amount of gas that can be used to add transactions into a block.
We have also analyzed how different types of transactions can cause different performance results in the network.
References:
Although this article is mainly focused on private ethereum networks, there exist great articles that covers similar topics but for public networks, some links here:
Author: Erick Pacheco — Researcher at Innovation & Labs Blockchain/DLT Peru in everis
