IPFS Cluster with Docker

happy devOps

(λx.x)eranga

Published in

Effectz.AI

3 min readOct 16, 2020

Background

IPFS which know as Interplanetary File System is peer to peer decentralized storage system which can be used share files(data) through a distributed file system. It made up of geographically distributed nodes where people and apps can storing and sharing files. IPFS supports to store files and track versions over time, very much like git. It uses content-addressing to uniquely identify the data in the network. Each file saved in the IPFS identified with its CID(content identifier) which derived through the hash of the file.

Part of the IPFS data lifecycle is garbage collection. This is when a node deletes old or unused data from disk to help save space and keep the node efficient. IPFS nodes treat the data they store like a cache, meaning that there is no guarantee that the data will continue to be stored. To ensure data stays permanently in your local node, you can pin files to the node and stop them from throw into the garbage collection cycle. Pinning a CID tells an IPFS node that the data is important and should not be deleted. These pinned data made permanently available only in the pinned node unless other nodes fetched those data. It introduced single point of failure where node goes down data on that node may lost. To overcome this concern IPFS introduced IPFS Cluster which enables automatically replicating and pinning the data throughout the IPFS network of nodes. Whenever a new data is generated and pinned all the cluster nodes receive it. The pin replication handles by the IPFS cluster. It prevent the single point of failure in IPFS. IPFS Cluster is a distributed application that works as a sidecar to IPFS nodes. It runs alongside IPFS node as a separate service and needs to be installed on all the IPFS nodes that make your cluster.

In this post I’m goanna discuss about running IPFS Cluster with Docker. All the source codes which related to this post available in gitlab. Please clone the repo and continue the post.

Cluster architecture

In this scenario I’m gonna run three node IPFS Cluster. It contains three IPFS nodes along with three IPFS Cluster nodes. Following figure described the architecture of this IPFS cluster.

Docker compose

I’m gonna deploy the IPFS cluster with docker-compose. Following is the docker-compose.yml deployment file I have used to run the cluster setup.

I can start the IPFS Cluster as a daemon with docker-compose up -d command. It will starts three IPFS nodes and three IPFS Cluster nodes.

Manage Cluster

There is a command line cluster management tool ipfs-cluster-ctl. It can be used as the client application to manage the cluster nodes and perform actions such as inspect the cluster, add and pin content. ipfs-cluster-ctl uses the HTTP API provided by the cluster nodes and it is completely separate from the cluster service. Download the ipfs-cluster-ctl from here into your local machine. It comes with ipfs-cluster-ctl shell script. Following are the various operations that can be performed with ipfs-cluster-ctl client. By default it connects to localhost cluster peers. The (--host) can be used to talk to any remote cluster peer. Since ipfs-cluster-ctl using HTTP API to serve the requests, we can view the HTTP request/response data of ipfs-cluster-ctl command in debug mode with ipfs-cluster-service --enc=json --debug <command>.

Since IPFS Cluster exposes REST API, we can directly interact with this REST API to mange the cluster as well(instead of ipfs-cluster-ctl). The IPFS Cluster REST API specification available in here. Following is the way to interact and mange the IPFS Cluster with the REST API via CURL. By default Cluster REST API runs 9094 port on the cluster nodes.