Hyperledger Caliper —Hyperledger Fabric TPS identification
Hyperledger Caliper is a platform provided by the Hyperledger project that facilitates the benchmarking of various blockchain platforms such as
- Hyperledger Besu
- Hyperledger Burrow
- Ethereum
- Hyperledger Fabric
- FISCO BCOS
- Hyperledger Iroha
- Hyperledger Sawtooth.
Hyperledger Caliper currently supports the following indicators resource consumption, transaction/read latency, transaction/read throughput and success rate.
In this blog we will discuss the lifecycle of hyperledger caliper, how to install hyperledger caliper, how to configure caliper with fabric and finally how to run a demo project. We hope that you have a basic knowledge of hyperledger fabric and quite familiar with the terms such as chaincode, peers, orderers, crypto-materials, etc.
Lifecycle of Hyperledger Caliper
The general life cycle of hyperledger caliper is as follows:
- Start SUT (System Under Test)
- Initialize SUT
- Install smart contracts/chaincode
- Perform benchmark test
- Tear down SUT
The above mentioned are the general steps and it is not necessary that we go through each step, ie: we can skip a step or two based on our requirements for eg: caliper-flow-skip-start — This skips the start of SUT
Installing Hyperledger Caliper
Hyperledger caliper is published as a node package @hyperledger/caliper-cli and also as a docker image hyperledger/caliper. In this blog we will be using the node package.
There are basically two ways to install this package:
- Globally:
- npm i -g — only=prod @hyperledger/caliper-cli@0.3.0
- Locally in a Repository:
- npm init -y
- npm i -g — only=prod @hyperledger/caliper-cli@0.3.0
After we have installed the package then we bind our to our required SDK package. Here we are binding it with fabric 1.4.6.(If we have installed globally we don’t need to use the npx prefix as shown below we can directly use caliper)
- npx caliper bind — caliper-bind-sut fabric — caliper-bind-sdk 1.4.6
This process of binding can also be done automatically when we are running the benchmark/testing process.
Running a demo project
The hyperledger caliper project repo comes with in build examples like the fabric samples. In this section we are going to run a demo project that has the following configuration solo orderer, 2 organizations with a peer each and 2 ca respectively. The project is a 1.4.1 version of fabric. So we have to bind the caliper accordingly.
The command to run the demo is very simple and is given below.(The below command i binded SUT SDK along with the launch command)
- npx caliper launch master — caliper-bind-sut fabric:1.4.0 — caliper-workspace . — caliper-benchconfig benchmarks/scenario/simple/config.yaml — caliper-networkconfig networks/fabric/fabric-v1.4.1/2org1peergoleveldb/fabric-go.yaml
Running this command we will start the demo project and run the test based on the config.yaml file inside the benchmarks/scenario/simple folder and the network configuration can be found in networks/fabric/fabric/fabric-v1.4.1/2org1peergoleveldb/fabric-go.yaml
So in the next section we will be discussing more on how to develop our own network configuration and config.yaml.
How to configure caliper with fabric project
Before we go in detail with configuring caliper and fabric, lets just go back to the demo we just ran and see what are the different things that are mentioned in those and get more clarity on things. Lets first analyze fabric-go.yaml file
The fabric-go.yaml file is the file that actually connects the caliper with our fabric network. If we browse through this file we can find some similarity with the network parameters of fabric. We basically configure the crypto details of the network along with the peer configuration and chaincode version and language. Etc
The config.yaml file is the file that actually where we define the number of rounds i.e. the number of different chaincode functions to be tested.
In each round we define the txNumber which says the number of transactions to be executed. We also define the tps (transaction per second) number of transactions to be sent in a second or txDuration which specifies the duration till which we will be sending the transaction. This is called rate controllers. There are different types of rate controllers:
- Fixed rate: The fixed rate controller is the most basic controller, and also the default option if no controller is specified. It will send input transactions at a fixed interval that is specified as TPS (transactions per second).
- Fixed feedback rate: The fixed feedback rate controller which is the extension of fixed rate also will originally send input transactions at a fixed interval. When the unfinished transactions exceed times of the defined unfinished transactions for each client,it will stop sending input transactions temporarily by sleeping a long period of time.
- Fixed backlog: The fixed backlog rate controller is a controller for driving the tests at a target loading (backlog transactions). This controller will aim to maintain a defined backlog of transactions within the system by modifying the driven TPS. The result is the maximum possible TPS for the system whilst maintaining the backlog level.
- Composite rate: A benchmark round in Caliper is associated with a single rate controller. However, a single rate controller is rarely sufficient to model advanced client behaviors. Moreover, implementing new rate controllers for such behaviors can be cumbersome and error-prone. Most of the time a complex client behavior can be split into several, simpler phases. Accordingly, the composite rate controller enables the configuration of multiple “simpler” rate controllers in a single round, promoting the reusability of existing rate controller implementations. The composite rate controller will automatically switch between the given controllers according to the specified weights
- Linear rate: Exploring the performance limits of a system usually consists of performing multiple measurements with increasing load intensity. However, finding the tipping point of the system this way is not easy, it is more like a trial-and-error method. The linear rate controller can gradually (linearly) change the TPS rate between a starting and finishing TPS value (both in increasing and decreasing manner). This makes it easier to find the workload rates that affect the system performance in an interesting way. The linear rate controller can be used in both duration-based and transaction number-based rounds.
The next important thing in a config file is the callback. The callback actually specifies a JavaScript file which will be called in each round execution. This JavaScript file has majorly three functions init, run and end. The init function is the first function that gets called upon the round starting and retrieves all the arguments that are passed from the config file like txNumber/txDuration arguments etc. After the init function is executed the run function is invoked and inside this function we call the invokeSmartContract() which takes context, chaincode name,version arguments and timeout. After the execution of the run function the end function is called to generate the report of the test.
For each chaincode invocation if we need to pass different arguments then we need to write a corresponding logic in this file using a function and call that function at the start of the run function. Similarly if we have a round of querying data from blockchain then we will probably query based on index or key so we need to write a logic to store these keys or indexes in an appropriately and export it so as to use it another round’s callback.
The below given is an example of code is given below:
Config file
Workload Modules/Callback
Network config file
Once the test is completed the caliper will generate a detailed report in html format in the base directory of the project which include all the parameters we discussed earlier.
Thank you for your valuable time for reading through this blog, I hope you have a basic understanding of hyperledger caliper and its uses along with the way to configure it fabric and test. Hope come back more interesting and fun articles until then
Peace Out ✌
