Hyperledger Fabric performance benchmarking using Hyperledger Caliper

1. Overview

In this article, we will be going to learn how we can do the performance benchmarking of Hyperledger Fabric blockchain with the help of Hyperledger Caliper. Caliper is a blockchain performance benchmark framework, which allows users to test different blockchain solutions with predefined use cases, and get a set of performance test results. In the end, the caliper generates the report in the form of an HTML document.

2. Caliper

Hyperledger Caliper is a blockchain benchmark tool, it allows users to measure the performance of a blockchain implementation with a set of predefined use cases. Hyperledger Caliper will produce reports containing a number of performance indicators to serve as a reference when using the following blockchain solutions: Hyperledger Besu, Ethereum, Hyperledger Fabric, FISCO BCOS.

Currently, caliper supports the following blockchain solutions:

• Hyperledger Besu, utilizing the Ethereum adapter.
• Hyperledger Fabric v1.X, v2.X
• Ethereum
• FISCO BCOS

With Caliper you can get the following performance indicators:

• Success rate
• Transaction/Read throughput
• Transaction/Read latency (minimum, maximum, average)
• Resource consumption (CPU, Memory, Network IO, …)

3. Prerequisite

1. A running HLF network (refer to this or this).
2. NodeJS must be installed (I tested this on node version 10,12 and 14).

4. Setup

1. Clone the repository

$ git clone https://github.com/adityajoshi12/fabric-samples-advance-topics.git
$ cd fabric-samples-advance-topics/test-network
$ ./network.sh up createChannel -ca -c mychannel  -i 2.2 -s couchdb
$ ./network.sh deployCC -ccn basic -ccv 1

2. To set up the caliper let's start with creating the folders, we will create a folder with the name caliper at the root level of the downloaded repository, after that, we need to create 3 folders inside the caliper directory.

$ cd caliper
$ mkdir networks benchmarks workload
$ npm init -y
$ npm install --only=prod @hyperledger/caliper-cli@0.4.2
$ npx caliper bind --caliper-bind-sut fabric:2.2

5. Network Configuration

Now we will create the network configuration file which is required by the caliper to interact with the blockchain network. We will create a file networkConfig.yaml inside the network folder that we created in the above step (make sure you are passing the correct certificate path).

6. Benchmark Configuration

After that, we will create our benchmark configuration file and I will keep all my benchmark configurations inside the benchmark folder. I will create the file with the name assetCCBenchmark.yaml .
The benchmark configuration file defines the benchmark rounds and references the defined workload module(s). It will specify the number of test workers to use when generating the load, the number of test rounds, the duration of each round, the rate control applied to the transaction load during each round, and options relating to monitoring. For this tutorial, I have included the monitoring of the docker containers using the docker module.

• test: The root level block that contains the benchmark test information.
• name: The name of the test, in this case, “basic-contract-benchmark”.
• description: A description for the benchmark, in this case, “A test benchmark”.
• workers: A set of keys used to define the number of workers (separate worker-client instances) used in the subsequent benchmark.
• rounds: An array of distinct test rounds that will be progressed sequentially. Rounds may be used to benchmark different smart contract methods or the same method in a different manner.

Each round block contains the following:

• label - the unique header label to use for the round.
• description - a description of the round being run.
• txDuration - the specification of the test duration, in seconds.
• txNumber — the number of transactions to perform.
• rateControl - a rate control type, with options.
• workload - the workload module to use, with arguments to pass to the module. All arguments passed are available as roundArguments within the workload module.

There are multiple types of rate controllers like fixed-load, fixed-rate, fixed-feedback-rate, linear-rate, etc

7. Creating Test Workload Module

The workload module interacts with the deployed smart contract during the benchmark round. The workload module extends the Caliper class WorkloadModuleBase from caliper-core. The workload module provides three overrides:

• initializeWorkloadModule - used to initialize any required items for the benchmark
• submitTransaction - used to interact with the smart contract method during the monitored phase of the benchmark
• cleanupWorkloadModule - used to clean up after the completion of the benchmark

Now we need to create a createAsset module which will create the assets in the blockchain, we are using the random data, you can have your own implementation. We need to pass the following details to sendRequests :

• contractId — the name of the smart contract that is to be used and is present within the Caliper network configuration file.
• contractFunction — the specific function within the smart contract to invoke.
• contractArguments — the arguments to pass to the smart contract function.
• invokerIdentity — the identity to use that is present within the Caliper network configuration file. This can be optional and caliper will select an identity for you (from the appropriate invoking organization or default organization) and in this tutorial, there would only ever be 1 identity to pick but for completeness, the examples explicitly define the identity.
• readOnly — if performing a query operation or not.

In a similar way let's create the readAsset module for reading the data from the smart contract. We are creating some data in the initializeWorkloadModule , inside submitTransaction we are reading the data from the smart contract and in the cleanupWorkloadModule we are deleting all the assets that we created in the initializeWorkloadModule .

8. Run the Caliper Benchmark

Now we are ready to run the performance benchmark using the configuration files and test module. The performance benchmark will be run using the Caliper CLI, which will need to be supplied a path to the workspace and workspace relative paths to the network configuration file and the benchmark configuration file. This information is provided with the flags --caliper-workspace, --caliper-networkconfig, and --caliper-benchconfig respectively.

Since the smart contract has already been installed and committed, Caliper only needs to perform the test phase. This is specified by using the flag --caliper-flow-only-test.

As we are using a 2.2 SUT, we must specify to use the gateway capability because there is no non-gateway version for the 2.x connectors, so specify the flag --caliper-fabric-gateway-enabled

// ensure that you are in caliper folder
$ npx caliper launch manager --caliper-workspace ./ --caliper-networkconfig networks/networkConfig.yaml --caliper-benchconfig benchmarks/assetCCBenchmark.yaml --caliper-flow-only-test --caliper-fabric-gateway-enabled

9. Caliper Configuration (Optional)

In step 8 we learned, how we can run the caliper tests, but the command used is too long, this can be improved by using the configuration file where we can put all of these parameters in the file and use it with caliper-cli.

For this, we need to create one more file caliper.yaml inside the caliper folder.

and finally, we can run it using the command.

$ npx caliper launch manager --caliper-fabric-gateway-enabled

10. Benchmark Results

The resulting report will be generated as report.html and can be viewed in the browser. The following items for each benchmark round:

• Name — the round name from the benchmark configuration file
• Succ/Fail — the number of successful/failing transactions
• Send Rate — the rate at which caliper issued the transactions
• Latency (max/min/avg) — statistics relating to the time taken in seconds between issuing a transaction and receiving a response
• Throughput — the average number of transactions processed per second

11. Summary

In this article, we learned how we can do testing of our hyperledger fabric application using the caliper. Source code can be found here

If you find this article helpful do hit the clap button and follow me for more such informative articles.

You can find me on Linkedin or stalk me on GitHub? If that’s too social for you, just drop a mail to adityaprakashjoshi1@gmail.com if you wish to talk tech with me.

Join Coinmonks Telegram Channel and Youtube Channel learn about crypto trading and investing