Multi node IPFS Cluster on Docker

happy devSecOps

(λx.x)eranga

Published in

Effectz.AI

4 min readMay 25, 2022

Background

In my previous post I have discussed about deploying IPFS Cluster with docker in a single host(I have deployed IPFS Cluster docker containers on single host). In this post I’m gonna discuss about deploying IPFS Cluster in multi host environment with docker. The deployments related to this post available on gitlab. Please clone the repo and continue the post.

Cluster architecture

I’m deploying three node IPFS Cluster. It contains three IPFS nodes along with three IPFS Cluster nodes. Following figure described the architecture of this IPFS cluster.

Bootstrap Peers

The IPFS peers configured with bootstrap list(list of peers) which the IPFS daemon learns about other peers on the network. IPFS comes with a default list of trusted peers, but we are free to modify the list to suit our needs. One popular use for a custom bootstrap list is to create a personal IPFS network. BUT, unlike the IPFS daemon, which by default connects to the public IPFS network and can discover other peers in it by first connecting to a well known list of available bootstrappers, a Cluster peer runs on a private network and does not have any public peer to bootstrap to. Thus, when starting IPFS Cluster peers for the first time, it is important to provide information so that they can discover the other peers and join the Cluster. I can modify the IPFS daemon bootstrap list with ipfs bootstrap add command. The IPFS Cluster bootstrap peers can be specified when starting cluster with --bootstrap <peer-multiaddress1,peer-multiaddress2> flag. In this cluster setup I’m using peer0 as the bootstrap peer of both IPFS and IPFS Cluster.

Cluster Secret

There is a 32-byte hex-encoded string(known as secret) which acts as libp2p network protector. This provides additional encryption for all communications between peers(libp2p) using a pre-shared key. Sharing the same cluster secret allow peers to understand that they are part of one IPFS-Cluster. Cluster secret makes it impossible to communicate with a peer’s swarm endpoint and thus, to send RPC commands to that peer, without knowing the secret in advance.

The secret is a security requirement for raft-based clusters which do not enforce any RPC authorization policy. CRDT-based clusters can run with an empty secret as long as trusted_peers is correctly set: only the peers in trusted_peers can modify the pinset and perform actions. However, it recommend to set the secret in all cases, as it provides network isolation: clusters running without a secret may discover and connect to the main IPFS network, which is mostly useless for the cluster peers (and for the IPFS network).

Cluster secret can be set with CLUSTER_SECRET environment variable. I have generated secret key with following command and set in each peer as an environment variable.

Deploy Peer0

Following is the docker-compose.yml deployment of peer0. It contains deployments IPFS container(ipfs0) and IPFS Cluster container(cluster0). This peer deployed in AWS instance with public IP address.

After deploying the container, I have connected to ipfs0 and get the IPFS address of that peer. This address is used as the IPFS bootstrap address in other peers. Then I have connected to cluster0 container and get the cluster address of that peer. This address is used as the IPFS Cluster bootstrap address for other peers(peer1 and peer2). I have set this address in the .env file of other peers(peer1 and peer2) as an environment variable CLUSTER_PEER0.

Deploy Peer1

Following is the docker-compose.yml deployment of peer1. It contains deployments IPFS container(ipfs1) and IPFS Cluster container(cluster1). This peer deployed in AWS instance.

Before deploying the peer1, I have added CLUSTER_PEER0 address into the .env file. This address passed into IPFS Cluster daemon --bootstrap command.

After deploying the ipfs1 container, I have connected to the container and added the ipfs0 peers address into the bootstrap list with ipfs bootstrap add command. Then this ipfs1 can find the other peers in the network via the bootstrap peer ipfs0.

Deploy Peer2

Following is the docker-compose.yml deployment of peer2. It contains deployments IPFS container(ipfs2) and IPFS Cluster container(cluster2). This peer deployed in the local machine.

Similar to peer1, before deploying the peer2, I have added CLUSTER_PEER0 address into the .env file. This address passed into IPFS Cluster daemon --bootstrap command.

After deploying the ipfs2 container, I have connected to the container and added the ipfs0 peers address into the bootstrap list with ipfs bootstrap add command. Then this ipfs2 can find the other peers in the network via the bootstrap peer ipfs0.

Test the Cluster

Finally I have connected to the IPFS cluster and pinned some CIDs in to the cluster. Following is the way to do the pinning CID and retrieve the pin statuses.