Blockchain Technology is gaining more attention day by day but there is no way you can test the performance of various available blockchain platforms before creating the solution for your business problem. To keep this issue in mind Hyperledger community came with a tool “Hyperledger Caliper” that can be used to measure blockchain performance.
Hyperledger Caliper
Caliper is a blockchain performance benchmark framework, which allows users to test different blockchain solutions with predfined use cases, and get a set of performance test results.
Currently supported performance indicators:
- Success rate
- Transaction/Read throughput
- Transaction/Read latency(minimum, maximum, average, percentile)
- Resource consumption(CPU, Memory, Network IO,…..)
Architecture
Adaptation Layer
The adaption layer is used to integrate existing blockchain system into Caliper framework.Each adaptor implements the Caliper Blockchain NBIs by using corresponding blockchain’s native SDK or RESTful API.
Interface & Core Layer
The interface & Core layer implements core functions and provides north bound interfaces for up-applications. Four kinds of NBIs are provided:
1. Blockchain operating interfaces: Contains operations such as deploying smart contracts on backend blockchain, invoking contracts, querying states from the ledger etc.
2. Resource Monitor: Contains operations to start/stop a monitor and fetch resource consumption status of backend blockchain system, including CPU, memory, network IO, etc. Two kinds of monitors are provided now, one is to watch local/remote docker container, and another is to watch local processes.
3. Performance Analyzer: Contains operations to read predefined performance statistics (including TPS, delay, success ratio etc) and print benchmark results. Key metrics are recorded while invoking blockchain NBIs, e.g. created time and committed time of the transaction, result of the transaction etc.
4. Report Generator: Contains operatios to generate a HTML format testing report
Application Layer
The application layer contains the tests implemented for typical blockchain scenarios. Each test has a configuration file which defines the backend blockchain network and test arguments.
A default benchmark engine is implemented to help developers to understand the framework and implement their own test quickly. Developers can use NBIs directly to implement their test without the framework.
Benchmark Engine:
Master:
The master implements a default test flow which contains three stages:
1. Preparing stage: In this stage, the master creates and initializes an internal blockchain object with the blockchain configuration file, deploys smart contracts as specified in the configuration and starts a monitor object to monitor the resource consumption of backend blockchain system.
2. Testing stage: Master starts a loop to perform tests according to the benachmark configuration file. Tasks will be generated and assigned to clients according to the defined workload. Performance statistic return by clients will be stored for later analyzing.
3. Reporting stage: Statistics from all clients of each test round are analyzed, and a HTML format report will be generated automatically.
Clients:
Local Clients:
In this mode, the master uses Node.js cluster module to fork multiple local clients to do the actual testing work.
Zookeeper Clients:
In this mode, multiple zookeeper client are launched independently. A zookeeper client will register itself after launch and watch for testing tasks. After testing, a znode which contains the result of performance statistics will be created.
Caliper Sample Report Format:
Configuration File:
Two kinds of configuration files are used. One is the benchmark configuration file, which defines the benchmark like workload.
Another is the blockchain configuration file, which specify necessary information to help interacting with the SUT.
Below is a benchmark configuration file example:
{
// blockchain defines the type of backend blockchain system and the configuration file for the adaptor to recognize the backend blockchain network with which to interact.
"blockchain": {
"type": "fabric",
"config": "./fabric.json"
},
//defines commands which will be called at particular phases of the test
"command" : {
"start": "docker-compose -f ../../network/fabric/simplenetwork/docker-compose.yaml up -d",
"end" : "docker-compose -f ../../network/fabric/simplenetwork/docker-compose.yaml down;docker rm $(docker ps -aq)"
},
// defines the metadata of the test, as well as multiple test rounds with specified workload
"test": {
"name": "simple",
"description" : "This is an example benchmark for caliper",
"clients": {
"type": "local",
"number": 5
},
"rounds": [{
"label" : "open",
"txNumber" : [5000, 5000, 5000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 100}}, {"type": "fixed-rate", "opts": {"tps" : 200}}, {"type": "fixed-rate", "opts": {"tps" : 300}}],
"arguments": { "money": 10000 },
"callback" : "benchmark/simple/open.js"
},
{
"label" : "query",
"txNumber" : [5000, 5000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 300}}, {"type": "fixed-rate", "opts": {"tps" : 400}}],
"callback" : "benchmark/simple/query.js"
}]
},
// defines the type of resource monitors and monitored objects, as well as the time interval for the monitoring.
"monitor": {
"type": ["docker", "process"],
"docker":{
"name": ["peer0.org1.example.com", "http://192.168.1.100:2375/orderer.example.com"]
},
"process": [
{
"command" : "node",
"arguments" : "local-client.js",
"multiOutput" : "avg"
}
],
"interval": 1
}
}Pre-requisites:
- Make sure following tools are installed”
- NodeJS 8.X
- node-gyp
- Docker
- Docker-compose
git clone https://github.com/hyperledger/caliper.gitRun npm install in caliper folder to install dependencies locally.
Install blockchain SDKs
Fabic
run npm install grpc@1.10.1 fabric-ca-client fabric-client in the root folder
Sawtooth
Install dependencies:
npm install protocol-buffers
Install sawtooth javascript sdk using repository
run npm install sawtooth-sdk in the root folder
Iroha
Install Iroha library
npm install –no-save iroha-lib@0.1.7 in Caliper’s root folder
Composer
Install dependencies
Update the main package.json file to include the required Compsoer and Fabic modules, and subsequently run an npm install.
Run benchmark
All predefined benchmarks can be found in benchmark folder.
To start your benchmark, just run this in root folder
node benchmark/simple/main.js -c yourconfig.json -n yournetwork.json
-c : specify the config file of the benchmark, if not used, config.json will be used as default.
-n : specify the config file of the blockchain network under test. If not used, the files address musr be specified in the benachmark config file.
Note:
You can also use npm scripts to run a benchmark.
npm run list
> caliper@0.1.0 list /home/hurf/caliper
> node ./scripts/list.js
Available benchmarks:
drm
simple
Run a benchmark with specific config files
npm test — — simple -c ./benchmark/simple/config.json -n ./benchmark/simple/fabric.json
> caliper@0.1.0 test /home/hurf/caliper
> node ./scripts/test.js "simple" "-c" "./benchmark/simple/config.json" "-n" "./benchmark/simple/fabric.json"
......Run benchmark with distributed clients
1. Start the ZooKeeper service
2. Launch clients on target machines separately by running node ./src/comm/client/zoo-client.js zookeeper-server or npm run startclient — — zookeeper-server
e.g.
$ npm run startclient — 10.229.42.159:2181
> caliper@0.1.0 startclient /home/hurf/caliper
> node ./src/comm/client/zoo-client.js "10.229.42.159:2181"Connected to ZooKeeper
Created client node:/caliper/clients/client_1514532063571_0000000006
Created receiving queue at:/caliper/client_1514532063571_0000000006_in
Created sending queue at:/caliper/client_1514532063571_0000000006_out
Waiting for messages at:/caliper/client_1514532063571_0000000006_in......
3. Modify the client type setting in configuration file to zookeeper
e.g.
"clients": {
"type": "zookeeper",
"zoo" : {
"server": "10.229.42.159:2181",
"clientsPerHost": 5
}
}4. Launch the benchmark on any machine as usual.
Note:
- Zookeeper is used to register clients and exchange messages. A launched client will add a new znode under /caliper/clients/. The benchmark checks the directory to learn how many clients are there, and assign tasks to each client according to the workload.
- There is no automatic time synchronization between the clients. You should manually synchronize time between target machines, for example using ‘ntpdate’.
- The blockchain configuration file must exist on machines which run the client, and the relative path (relative to the caliper folder) of the file must be identical. All referenced files in the configuration must also exist.
Write your own benchmakrs
Caliper provides a set of nodejs NBIs (North Bound Interfaces) for applications to interact with backend blockchain system. Check the src/comm/blockchain.js to learn about the NBIs. MultipleAdaptors are implemented to translate the NBIs to different blockchain protocols. So developers can write a benchmark once, and run it with different blockchain systems.
Generally speaking, to write a new caliper benchmark, you need to:
- Write smart contracts for systems you want to test
- Write a testing flow using caliper NBIs. Caliper provides a default benchmark engine, which is pluggable and configurable to integrate new tests easily. For more details, please refer to Benchmark Engine.
- Write a configuration file to define the backend network and benchmark argument
Reference: https://github.com/hyperledger/caliper
In the next article i will sharing the performance test detail tutorial for Fabric.
You can also explore the various tutorials written by me on Blockchain Technology.
