Decentralized File Storage using Web3.Storage, IPFS, Filecoin, and FairOS for a Fair Data Society!
GM, hey everyone, I’m Mohd Sameer, a Blockchain Developer at VideoWiki associated with GetBoarded Technology. Recently, I had the opportunity to work on an exciting project where I worked on IPFS storage, a decentralized storage solution, using web3.storage to publish metadata content. Also we are working on the integration of FairOS, where FairOS is a decentralized operating system that manages the resources of computers in its network for storage and compute on top of Swarm.
In this post, I want to share my experience with integrating IPFS storage using web3.storage and some insights that I gained along the way. So, if you’re interested in learning about this fascinating technology and how it can be used, read on, my ser!
In this blog, I will be discussing:
- What are decentralization and IPFS?
- What is FairOS, also what are the Fair Data Society principles that it follows.
- Integration and use case of FairOS in VideoWiki.
- What is web3 storage, how it works, and its use cases?
- How we implemented IPFS storage using web3.storage in VideoWiki
- A step-by-step guide to integrating IPFS storage using web3.storage with code snippets and link to the notion for testing out more features
- Conclusion to all our learnings
IPFS and Decentralised Network:
As Richard Hendricks portrayed the decentralization concept on a famous television show Silicon Valley, so are we still relying on centralized tech companies to store and manage our information? Or are you going to prefer decentralization to absorb the importance of its features and change the power dynamics between tech companies and their users?
So let’s assume, you want to store all your cat videos and images on a decentralized web. Hell yeah, what If I tell you that you can not only limit the storage of these files to one location but can explore the possibility to store this data that is spread out across a vast network of computer systems connected with each other via the internet.
So technically what we are doing to your “cat” is that we are running the cryptographic hash functions to produce a digest from the raw data and that in return gets us a Content Address/Identifier which helps the IPFS to locate this file or say your cat on the network. Also, don’t worry your cat is fine.
What are the hype and pros of using Web3.Storage and IPFS:
Web3.storage is the latest addition to the decentralized technology landscape, offering developers a seamless and frictionless experience when it comes to storing and accessing data on a decentralized network. This innovative platform harnesses the power of decentralized storage solutions, such as Filecoin and IPFS, to provide a secure and reliable solution for developers building next-generation applications and websites.
Under the hood, web3.storage is powered by the provable storage of Filecoin, ensuring that data is stored securely on the decentralized network. Additionally, data is made accessible to users over the public IPFS network, providing a robust and scalable solution for developers building decentralized applications.
IPFS (InterPlanetary File System) is a decentralized file storage system that allows for the storage and retrieval of data in a distributed network. IPFS aims to replace traditional centralized servers with a network of peer-to-peer nodes that can store and serve data in a more efficient and secure manner. IPFS is an identity protocol for storing and sharing data on the distributed web. It’s free and open-source.
Filecoin is a decentralized storage network that uses economic incentives to ensure reliable file storage. It operates on a peer-to-peer network, allowing anyone to participate as a storage provider or file owner. The pricing and availability of storage are determined by market forces, creating a fair and transparent system for file storage. Filecoin offers a secure and reliable alternative to traditional centralized storage solutions.
For more understanding of filecoin and IPFS visit the blog,
FairOS and Fair Data Society Principles:
FairOS is a decentralized operating system that enables users to run applications over the internet while retaining complete control over their data. This is achieved through the decentralization of both storage and compute resources, following the principles of Web 3.0 and fair data. FairOS consists of two layers, FairOS-dfs for storage and FairOS-compute for computing, and is designed to make building decentralized applications easier for developers.
FairOS-dfs utilizes Swarm as its base storage, allowing for censorship-resistant and decentralized data storage. Developers can also expect familiar storage tools similar to those used in Web 2.0 when building on FairOS. Currently, FairOS-dfs is in beta release, while FairOS-compute is still in the research phase.
Also we are aligning with the Fair Data Society principles:
These are listed here: https://principles.fairdatasociety.org/
Integration and Use Case of FairOS in VideoWiki:
FairOS-dfs, in combination with Swarm, provides a decentralized and secure solution for storing video content on the VideoWiki platform. By utilizing the decentralized storage capabilities of FairOS-dfs, VideoWiki offers users complete control over their data and an added layer of security against data breaches and censorship.
The working can be illustrated below:
FairOS-dfs is a storage solution designed specifically for the FairOS operating system. It is a lightweight, stateless layer that utilizes the foundational components of Swarm to provide advanced storage capabilities.
We’ll also be using Pods:
FairOS-dfs enables the creation of de-centralised logical drives called pods, which are associated with a new key pair generated using the user’s hd wallet. Each pod functions like a container that can store files, data, and related metadata, and is always under the control of the user who created it. Pods are used by Dapps to store user data separately, and can be easily created and organized by the user. Additionally, Public Pods allow for anonymous sharing of large datasets between FairOS-dfs users.
The data storage functionality of FairOS-dfs is leveraged by VideoWiki to provide users with a decentralized storage solution similar to Dropbox. This would allow users to store their videos and other content in a secure and decentralized manner, while retaining full ownership and control of their data.
In summary, integrating FairOS-dfs into VideoWiki’s platform allows for decentralized and secure storage of videos while providing users complete control and ownership of their data. By using FairOS-dfs, VideoWiki can offer a censorship-resistant platform for hosting videos without any limitations on content.
Our use case and what’s there for you:
We have implemented this feature in VideoWiki to publish the metadata on IPFS. Here are some potential use cases for this setup:
- Decentralized metadata content delivery: Metadata content can be stored on web3.storage and IPFS, allowing for decentralized and highly available content delivery.
- Tamper-proof data storage: Metadata content stored on web3.storage and IPFS is secured by cryptographic hashes, making it tamper-proof and ensuring the integrity of the stored content.
Overall, the combination of web3.storage and IPFS is a powerful solution for decentralized storage with improved scalability.
How we Implemented it in VideoWiki:
We Introduced web3.storage implementation, for our project. With web3.storage, there’s no need to worry about complex details or struggles. All we need to get started is a free API token.
So we have basically implemented this IPFS feature to publish our AI-generated video metadata content on the Filecoin network using our VideoWiki platform features.
The publishing mode on our platform lets us achieve this.
I have explained the work and implementation with all possible code snippets, just follow along and at the end of the blog, you’ll have all your files over the decentralized network.
PREREQUISITES:
Required Node version 14 or higher and NPM version 7 or higher. Check your local versions.
node --version && npm --version
> v16.4.2
> 7.18.1b
1. Setup:
Create a new account using GitHub or email to get the API Token and manage your data. This API Token lets you interact with web3.storage services.
Set up a new node project using npm init in order to work along the code snippets and explanations provided in the blog.
- Generate an API Key from web3.storage website, we’ll be needing this for storing the files.
API_KEY = <YOUR API KEY>- Package.json for the node file, copy and paste this into your project, and install these dependencies using npm or yarn.
{
"name": "web3-storage-quickstart",
"version": "0.0.0",
"private": true,
"description": "Get started using web3.storage in Node.js",
"type": "module",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"minimist": "^1.2.5", "web3.storage": "^3.1.0"
},
"author": "YOUR NAME",
"license": "(Apache-2.0 AND MIT)"
}- Download the dependencies above and web3.storage.
npm install
npm install web3.storage2. Upload Image and Metadata JSON:
- We need to get the image in the same directory, that is to be uploaded.
- Here we upload the image first and get its CDN.
- Then using the image CDN we create an Image URL using a gateway URL function we have made.
- Embed this URL in the metadata.json file that we create from objects to JSON using stringify
- Then we upload this JSON file to the IPFS and in return get its CDN and URL
- Now create uploadFunc.js, or whatever you want to name your js file. the code is well commented on and explains each line.
import { getFilesFromPath, Web3Storage } from "web3.storage";
import fs from "fs";
// import dotenv from 'dotenv'
// dotenv.config()
/* ********************** Token ********************** */
const apiToken = <YOUR API TOKEN>
/* ********************** Image ********************** */
/* Function gateway URL for image */
function makeGatewayURLImage(imgCID, imgName) {
return `https://${imgCID}.ipfs.w3s.link/${imgName}`;
}
/* Construct client with token and endpoint */
const client = new Web3Storage({ token: apiToken });
/* Get file from path for image */
const img = await getFilesFromPath("./img.jpg");
/* Get image name from path */
const imgName = img[0].name;
/* get imgCID from web3.storage */
const imgCID = await client.put(img, { name: "imgName" });
console.log("imgCID: ", imgCID);
/* Make gateway URL for image */
const imgURL = makeGatewayURLImage(imgCID, imgName);
console.log("imgURL: ", imgURL);
/* ******************** Metadata JSON ******************** */
/* Function for gateway URL for json */
function makeGatewayURLJSON(jsonCID, jsonName) {
return `https://${jsonCID}.ipfs.w3s.link/${jsonName}`;
}
/* metadata for json */
const metadata = {
name: "TestJSON",
description: "Uploading json with imgURL embedded",
image: imgURL,
attributes: [
{ trait_type: "Participant", value: "Amy" },
{ trait_type: "Event Name", value: "TestEvent" },
{ trait_type: "Date", value: "27-12-2022" },
],
};
/* conversion to json file using fs*/
fs.writeFileSync("metadata.json", JSON.stringify(metadata));
/* Get file from path for json */
const json = await getFilesFromPath("./metadata.json");
/* Get json name from path */
const jsonName = json[0].name;
/* get jsonCID from web3.storage */
const jsonCID = await client.put(json);
console.log("jsonCID: ", jsonCID);
/* Make gateway URL for json */
const jsonURL = makeGatewayURLJSON(jsonCID, jsonName);
console.log('jsonURL: ',jsonURL);- Run the upload javascript file code by
node uploadFunc.jsWe get something like this in the terminal, these are the respective file’s CDN and URLs generated for JSON Metadata and Image. The URLs can be used to access the files stored on the IPFS Network.
HURRAH AND KUDOS, you did it!
Now, If you’re curious about how to upload a video and multiple files in an array to IPFS you can refer to the below notion link for that, which contains a depth explanation with code snippets. Happy learning.
Notion link for uploading video and multiple files.
Conclusion:
In conclusion, web3.storage and IPFS offer a powerful and accessible solution for decentralized file storage. With web3.storage, users can enjoy the benefits of this technology without having to worry about the underlying complexities. Whether you’re a seasoned developer or just starting out, web3.storage makes it easy to store your files securely on the decentralized web.
Also, the integration of FairOS decentralised Storage system on top of Swarm nodes helps us and users achieve control over their data with Fair Data Society Principles.
Also, I would like to express my gratitude to Aman Bishnoi and Ritik Mahajan for their invaluable guidance and support throughout the integration process.
I would also like to thank the team at VideoWiki for providing me with this opportunity and for their support.
The Documentation of Web3.storage and FairOS are also a great help and I’d recommend following it since I followed it too and found it to be on point and super precise. For your convenience, we are attaching the link below:
Web3 Storage Documentation FairOS
