Setting up a Blockchain Business Network With Hyperledger Fabric & Composer Running in Multiple Physical Machine

THE ARCHITECTURE

Hyperledger Fabric has a set of components each with its own roles and functionalities. Each of these will be running on it’s own Docker instances and are configured to work together. These Docker instances even when running on multiple physical machines, can still communicate with each other. That’s essentially the crux of how a blockchain runs on several physical machines.

TERMS

Certificate Authority

The certificate authority is responsible for handling all the access control logic, issuing the identities and permission for the users in the Hyperledger blockchain network.

Orderer

In order to keep the entire network in a synchronized state, the orderer is used. Whenever a new transaction is to be committed, the orderer is the one informing all the peers about the transaction. A network can have multiple orderers, also it’s advised in order to maintain less faults.

Peers

Only peers are allowed to commit transactions in the business network. Also each peer has its own copy of the entire world state. It’s connected with CouchDB instances which acts as the database. An organisation can have multiple peers and one or more anchor peer will be used to communicate with other organisations.

In this example, we’ll be having 1 CA, 1 Orderer, and 3 Peers. With 2 Peers running in the first machine, and the other in a second machine.

PREREQUISITES

• Install Docker
• Install Composer
• Install Fabric Images (1.0.4)
• Install Fabric Tools (Use this)

SETTING UP

Since all the components are running inside individual Docker instances, we can easily configure them to run separately.

In this example, I’ll be taking the fabric-dev-server and modify it to run on multiple machines. This is mainly focused towards easier understanding and to use it like a boilerplate.

Here since we’re using only the hlfv1 we’re removing the hlfv11 folder and bringing all the contents of hlfv1 outside.

The Folder Structure

This is how your codebase should look,

Here if you notice, there are three new files docker-compose-peer2.yml, startFabric-Peer2.sh and stopFabric.sh. As the name implies these are related to the third peer which we will be running on a separate machine.

Peer Configurations

The default dev server comes with configuration for running one Peer. So, the certificates are created correspondingly. Since we have three peers in total, we need to generate certificates for all the three.

If you’ve installed the fabric tools with the above mentioned script, you’ll have configtxgen and cryptogen in your system. We’ll be using this to generate the certificates for our peers, and CA.

In cypto-config.yaml change the count under PeerOrgs → Template → Count to number of peers you want. In our case it is 3. Thus the contents of crypto-config.yaml looks like below.

OrdererOrgs:
  - Name: Orderer
    Domain: example.com
    Specs:
      - Hostname: orderer
PeerOrgs:
  - Name: Org1
    Domain: org1.example.com
    Template:
      Count: 3
    Users:
      Count: 0

And the configtx.yaml looks like,

---
Profiles:
    ComposerOrdererGenesis:
        Orderer:
            <<: *OrdererDefaults
            Organizations:
                - *OrdererOrg
        Consortiums:
            ComposerConsortium:
                Organizations:
                    - *Org1
    ComposerChannel:
        Consortium: ComposerConsortium
        Application:
            <<: *ApplicationDefaults
            Organizations:
                - *Org1
Organizations:
    - &OrdererOrg
        Name: OrdererOrg
        ID: OrdererMSP
        MSPDir: crypto-config/ordererOrganizations/example.com/msp
        AdminPrincipal: Role.MEMBER
    - &Org1
        Name: Org1
        ID: Org1MSP
        MSPDir: crypto-config/peerOrganizations/org1.example.com/msp
        AdminPrincipal: Role.MEMBER
        AnchorPeers:
            - Host: peer0.org1.example.com
              Port: 7051
Orderer: &OrdererDefaults
    OrdererType: solo
    Addresses:
        - orderer.example.com:7050
    BatchTimeout: 2s
    BatchSize:
        MaxMessageCount: 10
        AbsoluteMaxBytes: 98 MB
        PreferredMaxBytes: 512 KB
    Kafka:
        Brokers:
            - 127.0.0.1:9092
    Organizations:
Application: &ApplicationDefaults
    Organizations:

Once you made the modifications, run the following inside the composer folder in order to create the certificates for all the peers.

cd "$(dirname "$0")"
cryptogen generate --config=./crypto-config.yaml
export FABRIC_CFG_PATH=$PWD
configtxgen -profile ComposerOrdererGenesis -outputBlock ./composer-genesis.block
configtxgen -profile ComposerChannel -outputCreateChannelTx ./composer-channel.tx -channelID composerchannel

This creates all the certificates and the key under crypto-config folder. We will be using this to configure our Docker image’s environment variables.

CONFIGURING DOCKER SERVICES

In the docker-compose.yml we have added the following services.

• ca.org1.example.com
• orderer.example.com
• peer0.org1.example.com
• couchdb
• peer1.org1.example.com
• couchdb1

Here the peer0.org1.example.com uses couchdb as the world state database and peer1.org1.example.com uses couchdb1 as world state database.

In order to configure the Certificate Authority, we’ll be using the certificates that have been generated newly. In the command section of ca.org1.example.com make sure to use the proper private key file. It will be located under composer/crypto-config/peerOrganizations/

org1.example.com/ca/ Once you’ve updated it you’re good to go.

version: '2'
services:
  ca.org1.example.com:
    image: hyperledger/fabric-ca:$ARCH-1.0.4
    environment:
      - FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server
      - FABRIC_CA_SERVER_CA_NAME=ca.org1.example.com
#      - FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/org1.example.com-cert.pem
#      - FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/a22daf356b2aab5792ea53e35f66fccef1d7f1aa2b3a2b92dbfbf96a448ea26a_sk
    ports:
      - "7054:7054"
    command: sh -c 'fabric-ca-server start --ca.certfile /etc/hyperledger/fabric-ca-server-config/ca.org1.example.com-cert.pem --ca.keyfile /etc/hyperledger/fabric-ca-server-config/d4e3285d47260c640643feda80dd0f4ae76378f0047f1cfba3efca6555600b43_sk -b admin:adminpw -d'
    volumes:
      - ./crypto-config/peerOrganizations/org1.example.com/ca/:/etc/hyperledger/fabric-ca-server-config
    container_name: ca.org1.example.com
  orderer.example.com:
    container_name: orderer.example.com
    image: hyperledger/fabric-orderer:$ARCH-1.0.4
    environment:
      - ORDERER_GENERAL_LOGLEVEL=debug
      - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
      - ORDERER_GENERAL_GENESISMETHOD=file
      - ORDERER_GENERAL_GENESISFILE=/etc/hyperledger/configtx/composer-genesis.block
      - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
      - ORDERER_GENERAL_LOCALMSPDIR=/etc/hyperledger/msp/orderer/msp
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: orderer
    ports:
      - 7050:7050
    volumes:
        - ./:/etc/hyperledger/configtx
        - ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/etc/hyperledger/msp/orderer/msp
  peer0.org1.example.com:
    container_name: peer0.org1.example.com
    image: hyperledger/fabric-peer:$ARCH-1.0.4
    environment:
      - CORE_LOGGING_PEER=debug
      - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_PEER_ID=peer0.org1.example.com
      - CORE_PEER_ADDRESS=peer0.org1.example.com:7051
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb:5984
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: peer node start --peer-defaultchain=false
    ports:
      - 7051:7051
      - 7053:7053
    volumes:
        - /var/run/:/host/var/run/
        - ./:/etc/hyperledger/configtx
        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/peer/msp
        - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
    depends_on:
      - orderer.example.com
      - couchdb
  couchdb:
    container_name: couchdb
    image: hyperledger/fabric-couchdb:$ARCH-1.0.4
    ports:
      - 5984:5984
    environment:
      DB_URL: http://localhost:5984/member_db
  peer1.org1.example.com:
    container_name: peer1.org1.example.com
    image: hyperledger/fabric-peer:$ARCH-1.0.4
    environment:
      - CORE_LOGGING_PEER=debug
      - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_PEER_ID=peer1.org1.example.com
      - CORE_PEER_ADDRESS=peer1.org1.example.com:7051
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb1:5984
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: peer node start --peer-defaultchain=false
    ports:
      - 8051:7051
      - 8053:7053
    volumes:
        - /var/run/:/host/var/run/
        - ./:/etc/hyperledger/configtx
        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/msp:/etc/hyperledger/peer/msp
        - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
    depends_on:
      - orderer.example.com
      - couchdb1
  couchdb1:
    container_name: couchdb1
    image: hyperledger/fabric-couchdb:$ARCH-1.0.4
    ports:
      - 6984:5984
    environment:
      DB_URL: http://localhost:6984/member_db

Similarly we’ll create a second docker compose file for Peer 2.

version: '2'
services:
  peer2.org1.example.com:
    container_name: peer2.org1.example.com
    image: hyperledger/fabric-peer:$ARCH-1.0.4
    environment:
      - CORE_LOGGING_PEER=debug
      - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_PEER_ID=peer2.org1.example.com
      - CORE_PEER_ADDRESS=peer2.org1.example.com:7051
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb2:5984
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: peer node start --peer-defaultchain=false
    ports:
      - 9051:7051
      - 9053:7053
    volumes:
        - /var/run/:/host/var/run/
        - ./:/etc/hyperledger/configtx
        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer2.org1.example.com/msp:/etc/hyperledger/peer/msp
        - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
    depends_on:
      - couchdb2
  couchdb2:
    container_name: couchdb2
    image: hyperledger/fabric-couchdb:$ARCH-1.0.4
    ports:
      - 7984:5984
    environment:
      DB_URL: http://localhost:7984/member_db

CONFIGURING THE SCRIPTS

Now we’re ready to run the Fabric network. We’ll be using the fabric-dev-server’s scripts as the base script with a few modifications as mentioned below.

In the startFabric.sh file, by default the configuration for the first peer (peer0.org1.example.com) to join the channel is provided. Now we need to join the second peer (peer1.org1.example.com) to the same channel.

In order to do that we need to fetch the channel block in the second peer and then use that .block file to join the peer. The following code is used to join the peers to the channel in the network.

# Create the channel
docker exec peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c composerchannel -f /etc/hyperledger/configtx/composer-channel.tx
# Join peer0.org1.example.com to the channel.
docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer0.org1.example.com peer channel join -b composerchannel.block
# # Create the channel
docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer1.org1.example.com peer channel fetch config -o orderer.example.com:7050 -c composerchannel
# docker exec peer1.org1.example.com peer channel create -o orderer.example.com:7050 -c composerchannel -f /etc/hyperledger/configtx/composer-channel.tx
# # Join peer1.org1.example.com to the channel.
docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer1.org1.example.com peer channel join -b composerchannel_config.block

In the second machine, we’ll be using the same concept of fetching the channel block and joining the channel with the block file.

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer2.org1.example.com peer channel fetch config -o orderer.example.com:7050 -c composerchannel
docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer2.org1.example.com peer channel join -b composerchannel_config.block

PEER ADMIN CARD CREATION

Now we need to update the script to create the peer admin card for the network. With the default script createPeerAdminCard.sh we’ll add update the connection profile configuration to add the new peers.

cat << EOF > /tmp/.connection.json
{
    "name": "hlfv1",
    "type": "hlfv1",
    "orderers": \[
       { "url" : "grpc://localhost:7050" }
    \],
    "ca": { 
        "url": "http://localhost:7054", 
        "name": "ca.org1.example.com"
    },
    "peers": \[
        {
            "requestURL": "grpc://localhost:7051",
            "eventURL": "grpc://localhost:7053"
        }, {
            "requestURL": "grpc://localhost:8051",
            "eventURL": "grpc://localhost:8053"
        }, {
            "requestURL": "grpc://<Peer2-IP>:9051",
            "eventURL": "grpc://<Peer2-IP>:9053"
        }
    \],
    "channel": "composerchannel",
    "mspID": "Org1MSP",
    "timeout": 300
}

Also you need to update the PRIVATE_KEY path with the corresponding path to the private key of map keystore.

PRIVATE_KEY="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/keystore/<PRIVATE_KEY_NAME>

This is how your script should look,

#!/bin/bash
# Exit on first error
set -e
# Grab the current directory
DIR="$( cd "$( dirname "${BASH_SOURCE\[0\]}" )" && pwd )"
echo
# check that the composer command exists at a version >v0.14
if hash composer 2>/dev/null; then
    composer --version | awk -F. '{if ($2<15) exit 1}'
    if \[ $? -eq 1 \]; then
        echo 'Sorry, Use createConnectionProfile for versions before v0.15.0' 
        exit 1
    else
        echo Using composer-cli at $(composer --version)
    fi
else
    echo 'Need to have composer-cli installed at v0.15 or greater'
    exit 1
fi
# need to get the certificate 
cat << EOF > /tmp/.connection.json
{
    "name": "hlfv1",
    "type": "hlfv1",
    "orderers": \[
       { "url" : "grpc://localhost:7050" }
    \],
    "ca": { 
        "url": "http://localhost:7054", 
        "name": "ca.org1.example.com"
    },
    "peers": \[
        {
            "requestURL": "grpc://localhost:7051",
            "eventURL": "grpc://localhost:7053"
        }, {
            "requestURL": "grpc://localhost:8051",
            "eventURL": "grpc://localhost:8053"
        }, {
            "requestURL": "grpc://192.168.31.12:9051",
            "eventURL": "grpc://192.168.31.12:9053"
        }
    \],
    "channel": "composerchannel",
    "mspID": "Org1MSP",
    "timeout": 300
}
EOF
PRIVATE_KEY="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/keystore/7fe58742a0b6d1102c74293808f1736dea010d3451f9e1a804c0b86ecf90baa0_sk
CERT="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/signcerts/Admin@org1.example.com-cert.pem
if composer card list -n PeerAdmin@hlfv1 > /dev/null; then
    composer card delete -n PeerAdmin@hlfv1
fi
composer card create -p /tmp/.connection.json -u PeerAdmin -c "${CERT}" -k "${PRIVATE_KEY}" -r PeerAdmin -r ChannelAdmin --file /tmp/PeerAdmin@hlfv1.card
composer card import --file /tmp/PeerAdmin@hlfv1.card 
rm -rf /tmp/.connection.json
echo "Hyperledger Composer PeerAdmin card has been imported"
composer card list

That’s it, everything is set and ready to go. Now lets run the following scripts to start the Hyperledger Fabric network on the first machine.

./teardownFabric.sh && ./startFabric.sh && ./createPeerAdminCard.sh

Run the following command on the second machine,

./startFabric-Peer2.sh

If everything is working properly, you’ll notice clean logs and zero errors. Try running docker ps to see all of the services that are running on each of the machines.

In the first machine there are 6 services running.

And in the second machine there are two services running.

When you notice the logs, the peers syncs whenever you create asset of submit transactions. The peer0 will be getting the first transaction submit request and then the orderer distributes to other peers maintaining all the peers in sync.

Checkout this repo for the entire source code of the server : https://github.com/varun-raj/fabric-dev-servers-multipeer

Follow us for more Hyperledger related articles.