Deploy IPFS Cluster with Kubernetes
Happy devSecOps
Background
In my previous post I have discussed about deploying IPFS Cluster with docker. In this post I’m gonna discuss about deploying IPFS Cluster with Kubernetes. All the deployments which related to this post available in gitlab. Please clone the repo and continue the post.
Cluster Architecture
In this scenario I have used Minikube-based Kubernets cluster. Read more about configuring Kubernets with Minikube from here. I have run three node IPFS Cluster
on Kubernets. It contains three IPFS nodes
along with three IPFS Cluster nodes
. Following figure described the architecture of this IPFS cluster.
Cluster Configurations
IPFS Cluster uses two main configurations files
service.json
— containing the cluster peer configuration, usually identical in all cluster peersidentity.json
— containing the unique identity used by each peer
The identity.json
includes a base64-encoded private key and the public peer ID associated to it. This peer ID identifies the peer in the Cluster. When automating the deployment we can generate peer IDs and private keys manually beforehand and override them with CLUSTER_ID
and CLUSTER_PRIVATEKEY
environment variables. In here I have generated peer ID and base64-encoded private key with ipfs-key
command.
The service.json
file contains 32-byte hex-encoded secret
which acts as libp2p
network protector. This provides additional encryption for all communications between peers (libp2p) using a pre-shared key. The secret value can be override with CLUSTER_SECRET
environment variable. I have generated the cluster secret with following command.
Then I have added these configurations into Kubernetes ConfigMap
and Secret
. CLUSTER_SECRET
and CLUSTER_PRIVATEKEY
added into Secret
. CLUSTER_ID
added into ConfigMap
.
Bootstrap Scripts
I have created another Kubernets ConfigMap
with two shell scripts entrypoint.sh
and configure-ipfs.sh
. entrypoint.sh
enables hands-free bootstrapping of the ipfs-cluster cluster. configure-ipfs.sh
configures the ipfs
daemon with production values. For more information about configuring ipfs for production, see go-ipfs
configuration tweaks. These scripts used in Kubernets StatefulSet
object which defined the ipfs-cluster and ipfs deployments.
IPFS Cluster StatefulSet
IPFS Cluster container and IPFS container deployed in Kubernets StatefulSet. StatefulSet run 3 replicas of the pod. Single pod contains ipfs-cluster
and ipfs
containers. The configure-ipfs.sh
and entrypoint.sh
scripts used as the command in ipfs-cluster and ipfs containers. The data volumes of ipfs-cluster and ipfs containers defined as volumeClaimTemplates
. Read more about the StatefulSet configurations from here.
IPFS Cluster Service
The final step is to define the Service
which expose the IPFS Cluster endpoints to the outside the Pod
. Following is the Kubernets Service definition which exposes IPFS Cluster endpoints to outside as LoadBalancer
.
Deploy IPFS Cluster
Following is the way to deploy the cluster. As mentioned above I have used Minikube
based Kubernets Cluster in this example. When deploying in Minikube
based cluster the persistent volumes which created in the /data
directory will be created inside Minikube
. This /data
directory need to have read write permissions.
Test IPFS Cluster
I have used LoadBalancer
service to expose the IPFS cluster endpoints. To access LoadBalancer
service from host machine in the Minikube
based Kubernets cluster I have used minikube tunnel
. It opens EXTERNAL-IP
for LoadBalancer service. Then I can access the IPFS Cluster services from host machine via this EXTERNAL-IP
. IPFS Cluster HTTP API exposed in 9094
port. In my scenario, this port mapped in LoadBalancer via 32371
port. By default IPFS Cluster APIs can access only through the localhost
. To enable remote access, I have set the http_listen_multiaddress
field in the service.json
with 0.0.0.0
address("http_listen_multiaddress": "/ip4/0.0.0.0/tcp/9094"
). Following is the way to access IPFS Cluster services via HTTP API.
Reference
- https://cluster.ipfs.io/documentation/guides/k8s/
- https://medium.com/rahasak/ipfs-cluster-with-docker-db2ec20a6cc1
- https://cluster.ipfs.io/documentation/reference/configuration/
- https://cluster.ipfs.io/documentation/guides/security/
- https://medium.com/temporal-cloud/temporal-kubernetes-stack-scaling-ipfs-clusters-sharding-the-pinset-15e614ba3003
- https://discuss.ipfs.io/t/ipfs-cluster-ctl-127-0-0-1-connect-connection-refused/9197/4
- https://labs.eleks.com/2019/03/ipfs-network-data-replication.html