Graphene TPS measurement test
Hello, dear friends!
We have already disclosed that in the summer of 2017, the Karma development team decided to create its own blockchain based on the Graphene fork. We have compared several public blockchains existing at the time, and Graphene suited us best for the following reasons:
- 100,000 transactions per second (TPS) against 20–30 for Ethereum and 7 for Bitcoin;
- Graphene allows building apps using the most tested and the fastest C++ language;
- Graphene is still the most popular blockchain in the world in terms of the number of signed transactions, thanks to Bitshares, Steemit and Golos.
Definitely 100,000 TPS is a very good looking number declared by the developers of the Graphene core. Theoretically, the Karma core blockchain based on the fork of the BitShares code should reach similar indicators. But what would happen in reality? Let us try to measure the “real” performance of Karma’s blockchain network.
What do we really want to measure?
When we talk about measuring some characteristics of systems, the developers mean gauging the speed of a system’s fastest component. However, application developers are more interested in the performance of endpoint APIs. The Graphene API public JSON-RPC methods are based on HTTP/HTTPS or WS/WSS. So this will be the target of our tests.
Common technical principles (Karma)
One of the Graphene protocol’s advantages is a simple way of creating a lightweight wallet without the need to store a complete blockchain DB on a local storage. It is enough to have:
- A blockchain private key;
- Access to the protected storage on the local storage;
- Ability to execute commands for signing any transaction;
- Broadcasting a signed transaction to the Karma testnet.
Actually, the nodes that process transactions may or may not participate in the consensus. It does not matter. The nodes that are not in consensus are called api-nodes. They can be used just for balancing transactions. Therefore, we will check not the “internal” capabilities of the network, which can be “synthetic” and close to declared values. We will try to estimate the real metrics of our network by sending queries with signed transactions to random points on the testnet.
We will be honest with ourselves, as we will be conducting a real sampling of the obtained analysis results without using invasive methods.
Performance measuring method
We can make several assumptions:
- We have a running testnet with an assembled set of consensus;
2. We have N API-nodes, which will be connected with our clients and will send transactions;
3. The network is not be tuned in any way as we use the standard block closing times;
4. We assume that the end user (developer) is not familiar with C ++ and will try to develop his application for the network on one of the available tools:
- python
- node.js
5. We use a slightly modified library for python: https://github.com/bitshares/python-bitshares. The modification affects only the ability to pre-generate signed transactions with a deferred call. It needs to exclude the signing transaction time from total measuring time;
6. API-nodes are launched in a Docker instance on a local machine. It needs to exclude the time required for servicing TCP/IP connections;
7. All connections from clients are performed in random order for the selected address pool and random accounts of the Karma blockchain;
8. All transactions are the same in the form of token transfers between accounts;
9. Also, we assume that transactions can be repeated in random order by type of connection and the source of the connection;
10. Each transaction must end with a response from the node (we do not really care about the asynchronous behavior of the network);
11. We obtain two measurements:
- “Ecologically” clean, when one client generates only one transaction
- “Noisy”, when one client can generate many transactions per node, and in this case it is closer to the actual conditions for using network topology
12. Each test runs from 1 to N transactions for K available API-nodes;
13. Eventually, we compute a linear and logarithmic regression;
The Results
Fig. 1 shows the dependence of the overall network performance on the number of nodes involved in transaction processing. As you can see, the dependence curve is closer to a logarithmic form. Apparently, we have a tps limit close to 1600–2000.
Fig. 2 shows the dependence of the overall network performance on the number of nodes involved in transactions processing taking into account an arbitrary number of attempts (up to 20). In this case, the performance limit is a little less than in the previous test, but this is closer to real world conditions.
Finally Fig. 3 shows a comparative representation of dependencies under different testing conditions. We can assume that the curves converge at a single point.
Conclusions
The performance of the real network, that we can at least assemble “straight out the box”, is way far from the declared 100'000 tps. Perhaps, well-tuned nodes launched with a perfect network topology and very lightweighted smart contract will boost performance. But, in our case, using basic development tools we couldn't get even close to 10'000 tps.
Despite the outcomes, the BitShares engine still demonstrates relatively impressive results. This is certain relative to Bitcoin and Ethereum. As for the higher performance blockchain engines, either are still not available as products, or have not been submitted to any serious analysis yet.
Look forward to read more interesting tech-savvy articles from Karma team.
Cheers!
^_^
