Helm Chart for Fabric (for Kubernetes)

A Failure-Success

A summary of my disappointing failure attempting to create a Helm Chart deployment for Hyperledger Fabric (for Kubernetes). The purpose of this short series of stories is to document the achievements, describe the limitations and — hopefully — provide a coherent summary of the work to serve as the project’s documentation.

While Fabric is complicated to deploy across multiple hosts, I think the bulk of the challenges I’ve faced is in attempting to jam Fabric into a Helm Chart; I think Helm’s where I’m more challenged. I’m very grateful to Yacov and Gari at IBM for their ever-patient and helpful guidance on Fabric. Thanks both of you!

The Recommendation

I believe (strongly) that the Hyperledger Fabric project should commit to developing a working Helm Chart solution for Fabric. Fabric is a complex solution that changes significantly between its frequent releases. The core development team knows best how to configure and deploy the solution and is best able to keep the deployment solutions current with the core product.

Since Helm has become a de facto deployment tool for Kubernetes applications, I consider the Fabric team to be best-placed to identify Helm limitations and|or tweak Fabric to accommodate these.

The Success

It works ;-)

2x 2-Peer Orgs

And:

peer chaincode query \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["query","a"]}'
100
peer chaincode invoke \
--orderer=${RELEASE_NAME}-hyperledger-fabric-orderer:7050 \
--cafile=/.../example.com/orderers/orderer.example.com/... \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["invoke","a","b","10"]}'
2018-08-31 22:58:56.724 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 001 Chaincode invoke successful. result: status:200 
peer chaincode query \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["query","a"]}'
90

The Outstanding Problems

Unfortunately, to get chaincode instantiations to work requires manual intervention :-(

peer chaincode list --channelID=${CHANNEL_NAME} --installed
Get installed chaincodes on peer:
Name: duvall, Version: 1.0, Path: github.com/chaincode/example02/go/, Id: 4ed73815477c6da1a5e8fba4ff00d77cc740e12b2dd001246de4a53e0dc08c03

To instantiate chaincode, the peer uses docker-in-docker to create a Docker image of the chaincode deployment and then instantiates it:

docker images --format="{{.Repository}}" | awk '!/gcr.io/'
dev-org1-peer0-duvall-1.0-2c49d7...

NB The image name combines the network name (dev), the peer’s name (org1-peer0), the chaincode name (duvall) and version (1.0)

Monitoring Docker events for this image, we can detect when a container is created from the image and when the container is started:

docker events --filter=name=dev-org1-peer0-duvall-1.0-2c49d7...

NB The event output includes the long container ID (3e1b22…)

Before the container dies, we can grab its logs:

docker logs 3e1b22...

Which yields:

The issue arises becauses the container is run in the context of Docker Engine not “in” Kubernetes. It’s attempts to call back to the peer fail because Docker Engine is unable to address the peer.

Hack #1: Edit the Node’s Hosts File

The peer is available on 10.121.1.224for this deployment:

NB The address corresponds to the Cluster IP address for the Kubernetes Service that represents the peer.

Editing the Node’s (!) /etc/hosts permits Docker Engine to correctly address the Peer:

more /etc/hosts
127.0.0.1 localhost
::1  localhost
169.254.169.254 metadata.google.internal metadata
10.121.1.224 x-hyperledger-fabric-org1-peer0

Now, if we rerun the instantiation, it will work:

peer chaincode list --channelID=$CHANNEL_NAME --instantiated
Get instantiated chaincodes on channel channel:
Name: duvall, Version: 1.0, Path: github.com/chaincode/example02/go/, Escc: escc, Vscc: vscc

And, we can verify using the container’s logs:

docker logs 7e1a1e40e68b78653eb85103d0af03502820147ded1fa9288f70bf796b87ee37
2018-08-31 21:52:43.825 UTC [shim] SetupChaincodeLogging -> INFO 001 Chaincode (build level: 1.2.0) starting up ...
ex02 Init
Aval = 100, Bval = 200

It’s unclear to me how best to appropriately, dynamically ensure that every Nodes’ (plural) Docker Engine is able to correctly refer to the Peers (plural) that may be running on it at that instant.

Once instantiated, to invoke methods on the chaincode, the peer needs to be able to access the orderer:

peer chaincode query \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["query","a"]}'
100
peer chaincode invoke \
--orderer=${RELEASE_NAME}-hyperledger-fabric-orderer:7050 \
--cafile=/.../example.com/orderers/orderer.example.com/... \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["invoke","a","b","10"]}'
2018-08-31 22:58:56.724 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 001 Chaincode invoke successful. result: status:200 
peer chaincode query \
--channelID=${CHANNEL_NAME} \
--name=${NAME} \
--ctor='{"Args":["query","a"]}'
90

And, the container’s logs:

docker logs 7e1a1e...
2018-08-31 21:52:43.825 UTC [shim] SetupChaincodeLogging -> INFO 001 Chaincode (build level: 1.2.0) starting up ...
ex02 Init
Aval = 100, Bval = 200
ex02 Invoke
Query Response:{"Name":"a","Amount":"100"}
ex02 Invoke
Aval = 90, Bval = 210
ex02 Invoke
Query Response:{"Name":"a","Amount":"90"}

All good.

Hack #2: orderer.example.com

Except that it required a second hack.

This second issue is a consequence of the naming that’s defined for the network and expressed in crypto-config.yaml and configtx.yaml and (my) complexity in reflecting this appropriately in the cluster.

How does a Peer correctly address the Orderer when (my) Helm Chart results in the Orderer’s Service being available as e.g. x-hyperledger-fabric-orderer ([release-name]-[chart-name]-orderer) or fully-qualified as x-hyperledger-fabric-orderer.andromeda.svc.cluster.local?

The first hacky solution employed CoreDNS to provide example.com as a stub-domain (complementing Kubernetes’ on-cluster DNS resolution). My solution is over-engineered (see alternative below) but it provided me an opportunity to learn and use CoreDNS and in combination with Kubernetes. So a good learning experience and CoreDNS is a sweet product.

With this model, a Peer’s attempt to resolve e.g. orderer.example.com is shipped to CoreDNS and CoreDNS is programmed by the Kubernetes Services list to resolve orderer to the correct IP address.

I documented this approach here and won’t duplicate it.

The alternative (and much easier) hacky solution is to use Kubernetes’ hostAliases(link). These are defined as part of the manifest and results in the kubelet (?) programming the Pod’s (not the Node’s) /etc/hosts file.

Here’s an example manifest:

hostAliases:
- ip: "10.121.11.131"
  hostnames:
  - "orderer.example.com"

And here’s the Pod’s /etc/hosts:

more /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.56.0.9 x-hyperledger-fabric-org1-peer0-85874fd45c-nvrdc
# Entries added by HostAliases.
10.121.11.131 orderer.example.com

NB The addition of orderer.example.com and a useful system-provided comment explaining why it’s there.

In this approach, the Peer needs to be configured with the Orderer’s IP which I’m challenged to solve using Helm (;-().

The Implementation

I’ll briefly summarize my programming notes early next week and reference these here. Hopefully, a summary of my journey will provide others with some guidance on how they may proceed and avoid pitfalls. Hopefully, my mistakes will elicit feedback from others to help me improve and overcome some of the challenges I faced.

Conclusion

I sub-titled this story a “Failure-Success” because, while I’m able to deploy Fabric to a Kubernetes cluster, I was unable to develop a Helm Chart that (a) supported the dynamism I wanted; (b) provided an end-to-end working deployment.

As my manager correctly explained, it’s best to fail-fast and, while it’s disappointing that I was unable to finish this project, hopefully this and related stories will help others and perhaps it can find consensus within the Fabric project to develop a Helm Chart (or other Kubernetes deployment) for Fabric.

That’s it for now.