Deploy IPFS Cluster with Kubernetes

Happy devSecOps

(λx.x)eranga

Published in

Effectz.AI

3 min readApr 6, 2022

Background

In my previous post I have discussed about deploying IPFS Cluster with docker. In this post I’m gonna discuss about deploying IPFS Cluster with Kubernetes. All the deployments which related to this post available in gitlab. Please clone the repo and continue the post.

Cluster Architecture

In this scenario I have used Minikube-based Kubernets cluster. Read more about configuring Kubernets with Minikube from here. I have run three node IPFS Cluster on Kubernets. It contains three IPFS nodes along with three IPFS Cluster nodes. Following figure described the architecture of this IPFS cluster.

Cluster Configurations

IPFS Cluster uses two main configurations files

service.json— containing the cluster peer configuration, usually identical in all cluster peers
identity.json — containing the unique identity used by each peer

The identity.json includes a base64-encoded private key and the public peer ID associated to it. This peer ID identifies the peer in the Cluster. When automating the deployment we can generate peer IDs and private keys manually beforehand and override them with CLUSTER_ID and CLUSTER_PRIVATEKEY environment variables. In here I have generated peer ID and base64-encoded private key with ipfs-key command.

The service.json file contains 32-byte hex-encoded secret which acts as libp2p network protector. This provides additional encryption for all communications between peers (libp2p) using a pre-shared key. The secret value can be override with CLUSTER_SECRET environment variable. I have generated the cluster secret with following command.

Then I have added these configurations into Kubernetes ConfigMap and Secret. CLUSTER_SECRET and CLUSTER_PRIVATEKEY added into Secret. CLUSTER_ID added into ConfigMap.

Bootstrap Scripts

I have created another Kubernets ConfigMap with two shell scripts entrypoint.sh and configure-ipfs.sh. entrypoint.sh enables hands-free bootstrapping of the ipfs-cluster cluster. configure-ipfs.sh configures the ipfs daemon with production values. For more information about configuring ipfs for production, see go-ipfs configuration tweaks. These scripts used in Kubernets StatefulSet object which defined the ipfs-cluster and ipfs deployments.

IPFS Cluster StatefulSet

IPFS Cluster container and IPFS container deployed in Kubernets StatefulSet. StatefulSet run 3 replicas of the pod. Single pod contains ipfs-cluster and ipfs containers. The configure-ipfs.sh and entrypoint.sh scripts used as the command in ipfs-cluster and ipfs containers. The data volumes of ipfs-cluster and ipfs containers defined as volumeClaimTemplates. Read more about the StatefulSet configurations from here.

IPFS Cluster Service

The final step is to define the Service which expose the IPFS Cluster endpoints to the outside the Pod. Following is the Kubernets Service definition which exposes IPFS Cluster endpoints to outside as LoadBalancer.

Deploy IPFS Cluster

Following is the way to deploy the cluster. As mentioned above I have used Minikube based Kubernets Cluster in this example. When deploying in Minikube based cluster the persistent volumes which created in the /data directory will be created inside Minikube. This /data directory need to have read write permissions.

Test IPFS Cluster

I have used LoadBalancer service to expose the IPFS cluster endpoints. To access LoadBalancer service from host machine in the Minikube based Kubernets cluster I have used minikube tunnel. It opens EXTERNAL-IP for LoadBalancer service. Then I can access the IPFS Cluster services from host machine via this EXTERNAL-IP. IPFS Cluster HTTP API exposed in 9094 port. In my scenario, this port mapped in LoadBalancer via 32371 port. By default IPFS Cluster APIs can access only through the localhost. To enable remote access, I have set the http_listen_multiaddress field in the service.json with 0.0.0.0 address("http_listen_multiaddress": "/ip4/0.0.0.0/tcp/9094"). Following is the way to access IPFS Cluster services via HTTP API.