Why We Need Decentralized Storage
Currently, we are in the midst of a huge data explosion. Artificial Intelligence and Internet of Things devices are producing more amounts of data than we have ever seen. By the turn of the decade, a Gartner report estimates there will be more than 20 billion connected devices around the world.
The connected devices, combined with more applications, and the increase of data sharing between organizations is greatly increasing the need for more data storage solutions.
The drive towards more data-driven business is occurring at the same time as we have seen more data breaches than ever from centralized data centres.
A recent example of the damage that hacks can reek was the Equifax scandal in 2017. Social Security numbers, addresses, driver’s license numbers and birthdays were stolen from 145 million Americans and over 19000 Canadians who are now exposed to various types of fraud including false tax returns issued with their SIN or lines of credit opened in their names.
Other examples include the Anthem Health Insurance breach that exposed 78 million people alone or the massive Yahoo breach which affected over 1 billion yahoo accounts.
Hacking information is unbelievably lucrative and the larger the data set, the greater the opportunity which puts more and more organizations at risk for being targets past the usual large organizations. There are many uses for the information stolen or lost including insurance fraud, blackmail and even identity theft. Some hackers sell the information back to the affected organizations through ransomware and DDos attacks, others use the information themselves for fraud.
The data clearly shows that centralized databases, whether large or small, are extremely vulnerable.
Benefits of Decentralized Storage
Decentralized storage is a peer to peer network of pooled disk space that creates a shared universal memory. The idea is to use under-utilized drive capacity to create a storage marketplace that is more reliable and lower cost than cloud storage providers. Decentralized storage is close to what Dropbox offers, but without a central server.
Examples of decentralized storage solutions include IPFS or Sia.
Decentralized Storage solutions have a few specific characteristics that make it beneficial:
- Decentralized, no single party controls all the data
- Harder to Hack: Need to hack majority of nodes to get to the data
- No Outages: A power outage can greatly affect a centralized data centre- distributed storage solutions will not be hurt by a single outage to one node by a great degree
How It Works
So how does decentralized storage work?
Sharding
Sharding is basically a type of partitioning that separates large databases into smaller and easier-to-manage parts called shards. For a file, it would split the file into thousands of tiny pieces and store them separately on different nodes.
Sharding is very common place in centralized systems,but is still a new concept in decentralized system.
With decentralized storage, the shards are stored together and accessed by a decentralized application with a partition key.
Swarming
The storage of shards of data is done through swarming. Decentralized storage uses large groups of nodes (or swarms) to store the data. Shard teams or swarms can be created within a network where specific data is stored in a specific set of nodes based on some type of data, usually based on geography.
Swarming reduces latency and increases speed through retrieval of data from the fastest node; the nodes are vast and spread out all over the world creating increased scalability.
A major benefit of swarms is that individual nodes own the devices for storage, not single companies. This means customers looking for data storage have more options to choose from than the current major players.
The main issue is that the shard teams have to communicate with one another. If the shard teams don’t communicate well, bad actors can exploit the gap and potentially create problems.
In Blockchain
For a blockchain network, instead of storing the entire blockchain ledger on each node, the nodes are split up into shard teams where nodes only hold specific portions of the blockchain based on the group they belong to. This increases transaction efficiency within a blockchain.
A huge benefit of decentralized storage is that it will take the burden off of Blockchain networks. As blockchains begin to scale and handle increasingly more transactions, it will be unfeasible to store all the transaction data within the blockchain as it is not currently able to store large amounts.
Projects, such as FAB have already begun building blockchain networks which take advantage of other storage solutions in conjunction with Blockchain storage.
