Revoking and replacing an Intermediate CA in Hyperledger Fabric

Introduction

In Using 3rd Party Root CAs in Hyperledger Fabric, we learned how to provision a Fabric CA as an Intermediate CA (ICA), so that the Root CA (RCA) can be kept offline. In this article, you are going to perform certificate revocation and removal of an ICA from a network in the event that the ICA is compromised.

As a recap of the previous article, you have previously created two RCAs, one for identity certificates (Identity Root CA) and the other for TLS certificates (TLS Root CA). Each RCA issues an ICA certificate and the ICA deployed as Fabric CA instance hosts both certificate-key pair. Thus the ICA can issue Identity and TLS certificates for the peer and user.

Demo Scenario

A single-organization network which consists of a channel channel1 hosted by an orderer was also provisioned

Simple network

Note: Before proceeding with the steps below, ensure that you have followed all the steps in Using 3rd Party Root CAs in Hyperledger Fabric and have not destroyed the environment. The same pre-requisities apply. Additionally, ensure you have jq installed in your machine.

Revoke Identity ICA’s certificate

Let’s say for some reason, an unauthorized party gained access to the ICA’s Identity certificate-key pair. You need to make sure that the identities created by that compromised ICA cannot be used in the Hyperledger Fabric (HF) network. The first step is obviously to revoke the identity.

openssl ca -revoke crypto-config/peerOrganizations/org1.example.com/ca/ica.identity.org1.example.com.cert -config openssl_root-identity.cnf

Generate a Certificate Revocation List (CRL), which is a list of digital certificates that have been revoked by the issuing (CA) before their scheduled expiration date and should no longer be trusted. Store the CRL in the variable CRL as you will need it later.

openssl ca -gencrl -config openssl_root-identity.cnf -out identity-rca/crl/crls
export FLAG=$(if [ "$(uname -s)" == "Linux" ]; then echo "-d"; else echo "-b 0"; fi)
CRL=$(cat identity-rca/crl/crls | base64 $FLAG)

Create a replacement Identity ICA

Now that the ICA’s certificate is revoked, prepare a new ICA, newica.identity.org1.example.com, to take its place. First, generate certificate-key pairs and chain file for the new ICA. The steps are exactly the same as that of the original ICA

mkdir -p newica
openssl ecparam -name prime256v1 -genkey -noout -out newica/newica.identity.org1.example.com.key
openssl req -new -sha256 -key newica/newica.identity.org1.example.com.key -out newica/newica.identity.org1.example.com.csr -subj “/C=SG/ST=Singapore/L=Singapore/O=org1.example.com/OU=/CN=newica.identity.org1.example.com”
openssl ca -batch -config openssl_root-identity.cnf -extensions v3_intermediate_ca -days 1825 -notext -md sha256 -in newica/newica.identity.org1.example.com.csr -out newica/newica.identity.org1.example.com.cert
cat newica/newica.identity.org1.example.com.cert $PWD/identity-rca/certs/rca.identity.org1.example.com.cert > newica/newchain.identity.org1.example.com.cert

Start the new ICA.

docker-compose up -d newica.org1.example.com

Check that you can access the new ICA. Note that the port number is 8054 instead of 7054.

curl http://localhost:8054/cainfo\?ca\=ca

As usual, wait for a minute or two to ensure that the identities generated by the new ICA is not backdated. While waiting, let’s review our progress as shown in the figure below. Note that while the original ICA can issue both identity and TLS certificates, the new ICA can only issue identity certificates.

New ICA

Prepare replacement MSP administrator

Enroll the registrar of the new ICA

NEW_IDENTITY_REGISTRAR_DIR=crypto-config/peerOrganizations/org1.example.com/users/newadmin
mkdir -p $NEW_IDENTITY_REGISTRAR_DIR
export FABRIC_CA_CLIENT_HOME=$NEW_IDENTITY_REGISTRAR_DIR
fabric-ca-client enroll --caname ca --csr.names C=SG,ST=Singapore,L=Singapore,O=org1.example.com -m admin -u http://admin:adminpw@localhost:8054

The current Org1MSP administrator, Admin@org1.example.com will need to be removed as its certificate is issued by the compromised Identity ICA. Thus, a replacement administrator, newadmin@org1.example.com needs to be registered and enrolled

fabric-ca-client register --caname ca --id.name newadmin@org1.example.com --id.secret mysecret --id.type admin --id.affiliation org1 -u http://localhost:8054
NEWADMIN_DIR=crypto-config/peerOrganizations/org1.example.com/users/newadmin@org1.example.com
mkdir -p $NEWADMIN_DIR
export FABRIC_CA_CLIENT_HOME=$NEWADMIN_DIR
fabric-ca-client enroll --caname ca --csr.names C=SG,ST=Singapore,L=Singapore,O=org1.example.com -u http://newadmin@org1.example.com:mysecret@localhost:8054
cp newica/newchain.identity.org1.example.com.cert $NEWADMIN_DIR/msp/chain.cert
cp $PWD/nodeou.yaml $NEWADMIN_DIR/msp/config.yaml

Update channel configuration

In the blockchain network, Org1MSP is a member of channel1. A channel configuration update needs to be performed to remove and blacklist the compromised ICA. If Org1MSP is a member of more channels, the configuration update needs to be performed for all those channels. Fortunately, you only need to perform once in this demo scenario.

Load a helper script to help in performing channel configuration update. If jq is not installed, the following command will fail.

source utils.sh

The following activities will be performed as part of the channel configuration update:

1. Add the CRL into Org1MSP’s MSP definition
2. Replace the compromised ICA’s certificate (ica.identity.org1.example.com) with that of the new ICA (newica.identity.org1.example.com) in Org1MSP’s MSP definition
3. Replace the NodeOU identifiers with the new certificate chain (newchain.identity.org1.example.com.cert) in Org1MSP MSP definition

Channel Configuration Update

Technically speaking, replacing the compromised ICA’s certificate (Step 2) is enough to cause all identities issued by the compromised ICA to stop participating in the network. However, to prevent future accidential addition of the compromised certificate, the CRL needs to be included in the MSP definition (Step 1).

The first step is to retrieve the current configuration block of the channel

NEWICA=$(cat newica/newica.identity.org1.example.com.cert | base64 $FLAG)
NEWCHAIN=$(cat newica/newchain.identity.org1.example.com.cert | base64 $FLAG)
NEWADMIN=$(cat $NEWADMIN_DIR/msp/signcerts/cert.pem | base64 $FLAG)
WORKING_DIR=/config/channel1_update1
retrieve_current_config update1 channel1 Org1MSP \
/var/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp \
orderer.example.com:7050 \
/var/crypto/ordererOrganizations/example.com/msp/tlscacerts/tlsca.example.com-cert.pem

Insert the CRL:

docker exec -e "WORKING_DIR=$WORKING_DIR" -e "CRL=$CRL" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.revocation_list |= . + [\"$CRL\"]" $WORKING_DIR/current_config.json \
> $WORKING_DIR/tmp1_config.json'

Remove ica.identity.org1.example.com’s certificate:

docker exec -e "WORKING_DIR=$WORKING_DIR" cli \
sh -c 'jq "del(.channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.intermediate_certs[0])" $WORKING_DIR/tmp1_config.json \
> $WORKING_DIR/tmp2_config.json'

Add newica.identity.org1.example.com’s certificate:

docker exec -e "WORKING_DIR=$WORKING_DIR" -e "NEWICA=$NEWICA" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.intermediate_certs |= . + [\"$NEWICA\"]" $WORKING_DIR/tmp2_config.json \
> $WORKING_DIR/tmp3_config.json'

Replace the certificate chain in Fabric NodeOU:

docker exec -e "WORKING_DIR=$WORKING_DIR" -e "NEWCHAIN=$NEWCHAIN" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.fabric_node_ous.admin_ou_identifier.certificate = \"$NEWCHAIN\"" $WORKING_DIR/tmp3_config.json \
> $WORKING_DIR/tmp4_config.json'
docker exec -e "WORKING_DIR=$WORKING_DIR" -e "NEWCHAIN=$NEWCHAIN" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.fabric_node_ous.client_ou_identifier.certificate = \"$NEWCHAIN\"" $WORKING_DIR/tmp4_config.json \
> $WORKING_DIR/tmp5_config.json'
docker exec -e "WORKING_DIR=$WORKING_DIR" -e "NEWCHAIN=$NEWCHAIN" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.fabric_node_ous.orderer_ou_identifier.certificate = \"$NEWCHAIN\"" $WORKING_DIR/tmp5_config.json \
> $WORKING_DIR/tmp6_config.json'
docker exec -e "WORKING_DIR=$WORKING_DIR" -e "NEWCHAIN=$NEWCHAIN" cli \
sh -c 'jq ".channel_group.groups.Application.groups.Org1MSP.values.MSP.value.config.fabric_node_ous.peer_ou_identifier.certificate = \"$NEWCHAIN\"" $WORKING_DIR/tmp6_config.json \
> $WORKING_DIR/modified_config.json'

Prepare channel configuration update:

prepare_unsigned_modified_config update1 channel1

Finally, send the configuration update:

send_config_update update1 channel1 Org1MSP \
/var/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp \
orderer.example.com:7050 \
/var/crypto/ordererOrganizations/example.com/msp/tlscacerts/tlsca.example.com-cert.pem

To ensure that the channel update is successful, retrieve the updated configuration block:

retrieve_updated_config update1 channel1 Org1MSP \
/var/crypto/peerOrganizations/org1.example.com/users/newadmin@org1.example.com/msp \
orderer.example.com:7050 \
/var/crypto/ordererOrganizations/example.com/msp/tlscacerts/tlsca.example.com-cert.pem

If you are feeling curious, check the updated channel configuration in config/channel1_update1/updated_config.json

Replace peer’s certificate-key pair

Examine the peer’s log

docker logs peer0.org1.example.com

You should receive a similar log as the following. As the peer’s certificate is issued by the compromised ICA that has been removed, it no longer has access to retrieve the newer channel blocks.

...
2020-04-24 08:25:40.341 UTC [blocksProvider] DeliverBlocks -> ERRO 209 [channel1] Got error &{FORBIDDEN}
2020-04-24 08:25:43.515 UTC [deliveryClient] RequestBlocks -> INFO 20a Starting deliver with block [3] for channel channel1
2020-04-24 08:25:43.517 UTC [blocksProvider] DeliverBlocks -> ERRO 20b [channel1] Got error &{FORBIDDEN}
2020-04-24 08:25:49.925 UTC [deliveryClient] RequestBlocks -> INFO 20c Starting deliver with block [3] for channel channel1
2020-04-24 08:25:49.928 UTC [blocksProvider] DeliverBlocks -> ERRO 20d [channel1] Got error &{FORBIDDEN}
2020-04-24 08:25:59.936 UTC [deliveryClient] RequestBlocks -> INFO 20e Starting deliver with block [3] for channel channel1
2020-04-24 08:25:59.938 UTC [blocksProvider] DeliverBlocks -> ERRO 20f [channel1] Got error &{FORBIDDEN}

Try to invoke the chaincode.

docker exec -e "CORE_PEER_MSPCONFIGPATH=/var/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp" \
cli peer chaincode invoke -o orderer.example.com:7050 --tls \
--cafile /var/crypto/ordererOrganizations/example.com/msp/tlscacerts/tlsca.example.com-cert.pem \
-C channel1 -n chaincode1 -c '{"Args":["put", "y", "1"]}'

Not surprisingly, the operation will fail with an error message similar to the following:

Error: error sending transaction for invoke: got unexpected status: FORBIDDEN -- implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied:

However, query of keys inserted prior to the channel update still works, as those keys are already in the world state of the peer.

docker exec cli peer chaincode query -C channel1 -n chaincode1 -c '{"Args":["query","a"]}'

In order for the peer to be operational, the peer’s certificate needs to be replaced with one that is issued by the new ICA. First, stop the peer container.

docker stop peer0.org1.example.com

It is always a good practice to perform backup of the MSP folder

ORG_DIR=$PWD/crypto-config/peerOrganizations/org1.example.com
PEER_DIR=$ORG_DIR/peers/peer0.org1.example.com
mv $PEER_DIR/msp $PEER_DIR/msp-bak

Register and enroll a new peer identity, newpeer0.org1.example.com

export FABRIC_CA_CLIENT_HOME=$NEW_IDENTITY_REGISTRAR_DIR
fabric-ca-client register --caname ca --id.name newpeer0@org1.example.com --id.secret mysecret --id.type peer --id.affiliation org1 -u http://localhost:8054
export FABRIC_CA_CLIENT_HOME=$PEER_DIR
fabric-ca-client enroll --caname ca --csr.names C=SG,ST=Singapore,L=Singapore,O=org1.example.com -u http://newpeer0@org1.example.com:mysecret@localhost:8054
cp newica/newchain.identity.org1.example.com.cert $PEER_DIR/msp/chain.cert
cp $PWD/nodeou.yaml $PEER_DIR/msp/config.yaml

Start the peer.

docker start peer0.org1.example.com

Once again, examine the logs. The peer can now pull newer blocks from the channel

2020-04-24 08:38:47.477 UTC [gossip.privdata] StoreBlock -> INFO 034 [channel1] Received block [3] from buffer
2020-04-24 08:38:47.492 UTC [committer.txvalidator] Validate -> INFO 035 [channel1] Validated block [3] in 14ms
2020-04-24 08:38:47.509 UTC [kvledger] CommitWithPvtData -> INFO 036 [channel1] Committed block [3] with 1 transaction(s) in 16ms (state_validation=0ms block_and_pvtdata_commit=12ms state_commit=1ms) commitHash=[24daa73e029d52ab980ec0f9b770188535eba081d0ff4c39f0b6ace612229813]

Invoke will be successful, demonstrating that the peer is now operational

docker exec -e "CORE_PEER_MSPCONFIGPATH=/var/crypto/peerOrganizations/org1.example.com/users/newadmin@org1.example.com/msp" cli \
peer chaincode invoke -o orderer.example.com:7050 --tls \
--cafile /var/crypto/ordererOrganizations/example.com/msp/tlscacerts/tlsca.example.com-cert.pem \
-C channel1 -n chaincode1 -c '{"Args":["put", "y", "1"]}' --waitForEvent

Summary

We have demonstrated revoking and removal of an Identity Intermediate CA. The similar process applies for TLS Intermediate CA. I hope that you learned something new!