Using IPFS + FileCoin for decentralised storage with Powergate

Published in

Coinmonks

6 min readJun 29, 2020

I was working on designing a decentralised storage model using FileCoin for an application I am planning to work on during HackFS. Being new to Filecoin, I had some doubts regarding how to use it efficiently for my app’s storage needs.

For example, I found it difficult to come up with a storage deal based workflow for my application. Would users need to create a new storage deal for every new file upload? Do we need a subscription model? And how to renew storage deals approaching expiry?

This is when I came across Powergate by Textile. It seemed to answer much of the above questions through its FFS (Filecoin File Storage) module and I immediately wanted to test it first on an application that supported the following functions:

User login (creation of FFS instance per user)
Allow user to record audio
Save/delete recorded audio
Retrieve audio content back from IPFS and play.

Why this specific audio content based workflow? I’ll get to that and how these were implemented using Powergate later; first we need a brief overview of Powergate.

A (very) brief overview of Powergate

The Powergate is an API driven solution for deploying multitiered storage across Filecoin and IPFS. Persistent storage on Filecoin allows rich storage configuration for data such as replication factor, miner selection, deal renewal, and repair

A key point in the above definition is multitiered. Basically Powergate has hot and cold tiers of storage with IPFS nodes acting as the hot caching layer, and Filecoin serving as the cold persistance layer.

Storing new data through Powergate involves adding the data to the hot IPFS layer and push a config for the returned CID to Filecoin. The data identified by its CID can be retrieved back from IPFS through the Powergate client. In case the CID is unavailable on IPFS, the data can be retrieved from Filecoin automatically after a configured timeout.

This is a very brief overview. I’d encourage you to go through the Powergate documentation for a much better idea. I have listed some resources in the “Further Reading” section that helped me get started.

So how to use Powergate in an application?

It was relatively easy to come up with a basic design for this application once I went through the Powergate docs. The following diagram provides an overview of the design and the connected components.

High level design for an application using Powergate.

Let’s enlist the main interacting components from this design diagram.

A Svelte based frontend
Go backend
A PostgreSQL database
Powergate devnet instance (IPFS + Filecoin powered storage)

And as evident from the diagram, the frontend and the backend both need to interact directly with a Powergate devnet instance.

So why does the frontend need to interact with Powergate?

Persisting recorded audio tracks (upload data)
Retrieving audio tracks data to play (retrieve data)

But to make these upload and retrieval requests to FFS, we need to include an FFS token for authentication.

And where do we get this FFS token from?

The backend, on receiving an user login request, checks if any FFS token is associated with the user. If no such token exists, a request is sent to Powergate to create a new FFS instance along with its corresponding auth token. Update your user model on DB with this token and respond back to the login request with this token.

Creating new FFS instance

Since I use Go for this application backend, I used the Powergate Go client to create FFS instances. Since the frontend also makes requests to Powergate, we’ll also get to see how I used the JS client too in the next section.

Let’s check out a code block from the login handler that

sends a request to create an FFS instance, and
updates the user model with the token

if len(u.FFSToken) == 0 {
    // create a new FFS instance for user
    _, ffsToken, err := u.FFSCreate() // ignore instanceID for now
    if err != nil {
        // handle error
    }    u.FFSToken = ffsToken
    if err := h.userRepo.Update(u); err != nil {
        // handle error
    }
}

And this is how FFSCreate() looks like:

import (
    "github.com/multiformats/go-multiaddr"
    pow "github.com/textileio/powergate/api/client"
    "google.golang.org/grpc"
)func FFSCreate() (string, string, error) {
    ma, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/5002")
    if err != nil {
        // log error
        return "", "", err
    }    client, err := pow.NewClient(ma, grpc.WithInsecure())
    if err != nil {
        // log error
        return "", "", err
    }
    
    ffsID, ffsToken, err := client.FFS.Create(context.Background())
    if err != nil {
        // log error
        return "", "", err
    }    err = client.Close()
    if err != nil {
        // log error
    }    return ffsID, ffsToken, err
}

Okay, so first the powergate client needs to connect to the service API (default address at /ip4/0.0.0.0/tcp/5002 and hardcoded in the above code block).

Once the client is created, client.FFS.Create() does the job of creating an FFS instance and returning the ffsID and the ffsToken.

That’s it!

We can then update the user model and respond back to the login request with this token.

Persisting audio tracks

Once we have the FFS token, we can make authenticated requests to the FFS API.

Let’s first create powergate client and set the FFS token:

import { createPow } from "@textile/powergate-client";const pow = createPow({
    host: process.env.POW_HOST
});pow.setToken("some-ffs-token");

Now that the client is setup, let’s go ahead and store the data on IPFS+Filecoin. This happens in 2 steps —

Add data to IPFS cache and get CID (hot upload)

async function hotUpload(blob) {
    var audioBuffer = await blob.arrayBuffer();
    var input = new Uint8Array(audioBuffer);
    const { cid } = await pow.ffs.addToHot(input);
    return cid;
}

The blob already contains the recorded audio data that I convert to an Uint8Array to pass as an input parameter to pow.ffs.addToHot().

Push config corresponding to CID for persistence of data on Filecoin (cold upload)

async function coldUpload(cid) {
    const { jobId } = await pow.ffs.pushConfig(cid);
    return jobId;
}

This default config that we push corresponding to the cid here determines how we wish to persist the content on Filecoin.

For instance, we can configure whether the contents of the CID should be automatically retrieved after a specific configurable timeout from the cold tier if not available in the hot tier. Also, we can configure replication factor, storage deal duration and auto renewals. There are a bunch of other configurables that you can check out in the default config.

The jobId can be watched further for updates to the job status. Something like the following can be done to watch a jobId. (I myself haven’t been able to use it correctly so far due to my inadequacies wrt JavaScript)

// watch the FFS job status to see the storage process progressing
const cancel = pow.ffs.watchJobs((job) => {
  if (job.status === ffs.JobStatus.CANCELED) {
    console.log("job canceled")
  } else if (job.status === ffs.JobStatus.FAILED) {
    console.log("job failed")
  } else if (job.status === ffs.JobStatus.SUCCESS) {
    console.log("job success!")
  }
}, jobId)

Retrieving audio tracks from IPFS

Once an audio track is uploaded we would also need to retrieve the stored data to be able to play audio content. The following function getTrackContent() retrieves the data corresponding to a CID as an Uint8Array, creates a blob from it and returns the local URL that can be in turn passed to an audio element to be able to play the media.

async function getTrackContent(cid) {
    const audioBytes = await pow.ffs.get(cid);
    const audioBuffer = audioBytes.buffer;
    var blobOpts = { 'type' : 'audio/wav; codecs=0' };
    if (MediaRecorder.isTypeSupported("audio/wav;codecs=MS_PCM")) {
        blobOpts = { 'type' : 'audio/wav; codecs=MS_PCM' };
    }
    const blob = new Blob([audioBuffer], blobOpts);
    const url = URL.createObjectURL(blob);
    return url;
}

pow.ffs.get() tries to first retrieve the content corresponding to the CID from IPFS cache and if unavailable, gets the content from Filecoin after a confirgurable timeout.

All in all, it seems like Powergate actually does make life easy for application developers to start using Filecoin + IPFS for their decentralised storage needs. Looking forward to exploring it more during HackFS next month where I’ll be working on a collaborative music platform for amateur musicians looking for tracks to jam to!