How we were searching for an ideal DDS and Ignite technical overview
Our current project named Ignite is a decentralized social network, which allows everyone to share her/his mind freely via texts and media files. Ignute mission is to create a free flow of information online. It is a way to communicate globally that supports individual liberty of everyone without any form of censorship. Due to its decentralized nature Ignite is not governed by anyone and could not be blocked by any form of barrier or firewall.
Technically, it means that all the info within Ignite is distributed through blockchain and stored immutable in distributed data storage. We have been trying several distributed data storage services in the course of the development of Ignite.
A brief word on how Ignite works
Our architecture is the set of independent and equitable nodes, which exchanges every three minutes with the chunks of information containing all the user activities on the exact node for the three minutes. While creating this chunk, each node applies other nodes’ chunks to its local data storage. For user experience, this provides the common space with acceptable latency and minimal overheads.
From a practical standpoint that means we have a ton of chunks, which have been continuously uploading to the Arweave decentralized storage. Some of them may be huge but in fact a lot of them are small enough because they may contain info about just a couple of users’ comments and a dozen of posts’ likes (usual situation if it’s 3am for most of the node’s users). Thus, to make it moneywise we should be able to store a lot of files and most of them would be relatively small.
F(N) = N* 60/3 * P(a) * 24*30;
Where
F(N) is the number if the files for a month,
N is the number of the nodes,
P(a) is the probability of user activity on the node in 3 minutes.
What storage we have chosen and why
Actually, we have been trying most of the well-known blockchain based decentralized storages. We have been shifted from one to another for a bunch of reasons so we have got a clear enough picture about their practical effectiveness.
In the very beginning we started using Filecoin. It was back in March 2020. Filecoin is the creator of the Protocol Labs and actually the first and most famous blockchain based decentralized storage, “designed to store humanity’s most important information”.
It seems to fit us perfectly: to pursue policies of censorship resistance we do need the immutable and independent storage that will keep our most important humanity’s likes and comments. Filecoin looks really reliable — especially given the fact that it has attracted about 200,000,000$ during their tokensales and spent more than two years on system design and development.
However it seems that Filecoin development was taking way too long and keeping in mind all of its negative sides, such as minimal size requirement and quite high prices, we decided to abandon it.
Searching for DDS
After that, we’ve started our searches for an Ideal DDS of Ignite, we had a few in mind: Sia, Bittorrent File System (BTFS) and Arweave. Sia is the oldest decentralized data storage which we could be easy to integrate with, but the minimal requirements to the data chunk were also quite high. But after we’ve finished our research on SIA, we’ve realised that to ensure the best UI, we need to integrate with the CDN (Content Delivery Network), which also should be decentralized. There are not that many such solutions on the current market, so we decided to go with Skynet, created by Sia.
So we had only two left: BTFS and Arweave. BTFS was apparently cheaper to use, so we decided to try this one. We’ve done a full integration of BTFS to Ignite and were using it during the whole private beta stage. After the private beta was finished, we’ve realised that if we will continue to use BTFS as a primary DDS for Ignite, the total price for storage will snowball each month and reach around $10,000 per month already in December 2020. Also BTFS was quite raw for commercial use due to the bugs and shutdowns.
Why Arweave?
The only option we had was Arweave, so we decided to give it a try. At the first look Arweave storage prices were quite high compared to BTFS, but after we dipped into it, we realised that it’s way cheaper than other storages.
First of all, Arweave is a permanent storage, which takes the payment only once, which perfectly suits the Ignite model. To make it clear: Arweave uses initially paid AR tokens to generate the AR income, which is being used to extend the storage period.
The second great advantage is that arweave does not have a minimal requirement for the data chunk. Let us briefly explain what it means: the ordinary Ignite chunk size is relatively small, around 1 MB, so there is no point to pay for the whole chunk, which usually requires a minimum capacity of 10+ MB.
Utilizing Arweave allowed us to significantly reduce the storage expanses and create a sustainable storage model. As Ignite has to store all the information about posts, users, user actions etc. right from the private beta launch, monthly storage costs would be increasing rapidly, which would lead us, as a non-profit company, to very quick bankruptcy.
Another thing is that Arweave is one of the big players in the space, which provides sustainable technology. The Soter Gateway of BTFS which we used before was crashing every couple of days and caused serious injuries to the Ignite ecosystem.
We hope that in this article our community will find all answers to their recent questions, however if you still have something left, do not hesitate to pop-up in our official communities
Official website: https://prometeus.io
Ignite: https://ignite.so
Twitter: https://twitter.com/prometeusnet
Chat: https://t.me/promnetwork
Announcements: https://t.me/promnetwork_ann
Ignite official: https://beta.ignite.so/user/ignite_official
