Abundant Decentralized Storage on EOSIO
vRAM integrates RAM, IPFS, DISK and other systems into a single storage stack
TL;DR
- EOSIO developers have a range of storage systems at their disposal, each of which have their own unique strengths and tradeoffs.
- Instead of being yet another tier in this storage stack, vRAM brings it all together — as a paging mechanism for transferring data trustlessly among the different storage tiers.
- Thanks to vRAM, developers can combine the strengths of RAM, IPFS, and the recently-announced DISK database into a dynamic, affordable storage stack.
Just over a year ago, the vRAM system went live on the DAPP Network with the goal of solving one of the thorniest issues facing EOSIO devs: the scarcity of RAM. On the EOS mainnet, buying one megabyte of RAM — an EOSIO-specific database resource for storing dApp smart contracts and state information — currently costs a developer approximately 59 EOS. This makes it unsuitable for storing anything more than a basic amount of data. Other EOSIO chains offer more abundant resources today, but increased activity in the future may see their resource markets coming under similar strain.
For example, Wax provides 3kb of RAM for every new account. At the current capacity of 128GB, Wax would run out of RAM from accounts alone after 425,000 users — assuming RAM isn’t being used for anything else.
With vRAM, developers can store and retrieve exponentially greater amounts of data affordably and efficiently. vRAM works by connecting EOSIO RAM to IPFS, a decentralized storage solution with significantly more capacity than RAM. It serves as a paging mechanism that trustlessly relays data between IPFS and RAM, significantly reducing the RAM requirements for developers.
By using #vRAM to gain abundant storage resources, developers can future-proof their applications so that they will be ready to scale when adoption arrives. #DAPPNetwork (Click to Tweet)
Moonlighting, a freelancer platform with 750,000 users, has utilized vRAM to onboard their user base onto the EOS blockchain at 0.5% of the cost of the traditional EOS RAM model. Instead of storing user profiles in a limited RAM database, Moonlighting harnesses the abundant memory capacity of vRAM to store their data trustlessly and affordably.
Recently, the EOSIO community began to notice that a new storage resource called DISK is scheduled to be included in future versions of the software. According to Dan Larimer, Block.one’s CTO, “#eosio will soon have a disk backed key/value store as an option for contracts … This is especially useful for large datasets that are infrequently accessed.”
DISK allows developers to trade off speed for more capacity and could be incredibly synergistic with the other storage solutions available to EOSIO developers. Thanks to vRAM, DISK could be connected to RAM, IPFS, and other storage tiers to give developers more choice, more capacity, and more customization when building their dApps.
#vRAM brings together RAM, IPFS, DISK & other storage systems to form an abundant stack on #EOSIO.
It’s a trustless paging mechanism that allows devs to shuttle data between systems and achieve the perfect blend of affordability, speed & capacity. Click to Tweet
How vRAM Works
Before vRAM, dApp developers had to store both their smart contracts and any data associated with their contracts in a scarce RAM database. dApp state information, such as the balance for each user, would be stored permanently in RAM regardless of whether or not that user is currently interacting with the dApp. In that sense, EOS RAM is a misleading term, as it is at this time used more like a hard-disk drive rather than a random-access memory device that only stores data relevant to live operations.
vRAM frees up precious RAM resources by allowing dApp developers to store smart contract state data on chain history instead of directly in RAM. In addition, DAPP Service Providers (DSPs) also index the data by keeping a copy on their IPFS nodes. By hosting IPFS storage nodes in addition to their EOSIO nodes, DSPs provide developers with both redundant storage data and fast RAM warm up. Anytime the data is needed from within the smart contract, the DSP retrieves the data from its IPFS node and relays it to the smart contract, and a Merkle proof is used to validate its integrity.
The novelty of vRAM is that it connects between RAM and IPFS in a way that allows data to trustlessly move between these two storage tiers. It is like a shuttle route operated by the DSPs that transports data from IPFS into RAM when needed, and out of RAM as soon as the action, user session, or other relevant period is complete. As a result, EOSIO developers can harness a similar storage setup to that which computers use, shuttling data among storage tiers based on an application’s demands.
DISK is another stop on the route that data can travel. The route begins with RAM, which is quickest but most expensive, and goes all the way to the abundant yet slower IPFS storage. Developers need storage solutions that minimize latency and cost while maximizing capacity.
However, using a single tier of storage — be it IPFS, RAM, or DISK — involves making trade-offs between speed and cost-efficiency. Thanks to vRAM, developers can leverage multiple storage tiers together, shuttling information among them according to their changing needs.
A Tiered Storage Stack for EOSIO
RAM, IPFS, and now DISK are all different tiers on the storage stack available to EOSIO developers. With vRAM as the paging mechanism among the various tiers, developers can dynamically configure their storage stack to utilize the unique set of strengths and circumvent the trade-offs presented by each tier.
EOSIO’s tiered storage structure is not so different to traditional computing. A computer also consists of many storage tiers, each with its own unique strengths and trade-offs.
The bottom of a computer’s storage structure is closest to the processor and thus intensely expensive and limited but also intensely fast. As we move up the structure, the storage tiers increase in capacity and cost-effectiveness but are also orders of magnitude slower than the lower tiers.
At the bottom, we have CPU registers, which are the most fundamental component of a computer’s storage structure. Central Processing Unit registers can hold an extremely limited amount of data to facilitate some operation which takes place almost instantaneously. In addition to the registers, CPUs also have CPU caches whose purpose is to hold a temporary copy of data read from main memory. Caches are larger than registers, but also slower.
Next in line is a computer’s RAM, which serves as the computer’s whiteboard used to perform quick read and write operations. Data is fed into RAM from persistent storage devices, such as a computer’s internal drives (which themselves can be tiered, with faster SSDs and cheaper, slower HDDs), when it is needed for a specific operation. Once the operation completes execution, that section of RAM is free to be loaded by another program from the computer’s persistent storage. Finally, external hard drives and other backup devices or network solutions provide computers with much more storage capacity, yet reading from and writing to these devices is slower than accessing the computer’s native storage.
Similarly, the storage structure available to EOSIO developers follows a tier-based structure. RAM, at the bottom of the stack, is the quickest yet most costly way of storing dApp data. Higher up on the stack, IPFS is similar to external hard drives in that it offers abundant storage capacity but is significantly slower for developers building EOSIO dApps.
With DISK added to the picture, developers could now have more options at their disposal. DISK is a key-value pair store which may provide higher capacity than RAM but at the cost of more processing resources. While it is expected to be cheaper than RAM, DISK will also be slower and will require more CPU to run, which makes it less than ideal for storing data that is needed on-demand. On EOSIO’s storage stack, DISK falls in between RAM and distributed file storage systems, like IPFS, which are much more abundant than both RAM and DISK but also a slower than both.
Instead of being yet another tier in this storage stack, vRAM brings it all together as a paging mechanism for transferring data trustlessly among the different storage tiers. vRAM allows developers to optimize their applications by using different parts of the stack to balance cost-effectiveness, capacity, and speed.
Furthermore, vRAM works asynchronously, which means it doesn’t block the rest of the actions from executing on-chain. This makes access time less relevant since vRAM actions happen in parallel, unlike conventional paging mechanisms that create bottlenecks during execution.
Achieving Scalability Without Compromising on Decentralization
Not only does vRAM enable exponentially more memory capacity, it does so without compromising the base-layer scalability of EOSIO.
RAM, DISK and other base-layer resources are replicated across all the nodes in the system. Every single BP is required to support the resource requirements of the entire chain, which get more expensive as they grow. If the size of RAM on the network is allowed to increase continuously, it could create a barrier to entry for smaller BP teams that lack the ability to support such infrastructure. Eventually, block production would be centralized amongst the most well-capitalized teams who can afford infrastructure cost required to support the network.
In contrast, the vRAM System only replicates data storage on a subset of nodes — those belonging to the DSPs chosen by whomever is storing the data, allowing the system to scale horizontally.
Example vRAM dApp
To examine how vRAM could integrate DISK into the EOSIO storage stack, let’s walk through the architecture of EOSquare, a fictional social media dApp running on a public EOSIO chain.
This dApp would need to store data about every user and every post, including all comments and engagements, in a way that can be accessed from within the smart contract. Storing it all in RAM would be the quickest option, yet the scarcity of RAM resources make it infeasible to scale such a solution.
Instead, EOSquare can use RAM to store data only once a post or user is either created or engaged with, significantly reducing its RAM requirements. In the future, idle posts that were created recently could be moved onto DISK, where they would live until such time as someone engages with them. IPFS can serve as an archive for older posts and idle users who aren’t expected to be very active in the near future. Central to this multi-tiered storage architecture is vRAM, which allows data to flow freely among all of the tiers without compromising on decentralization.
A collaborative, ‘better together’ attitude compounds innovation and creates win-win situations for all.
It’s all too common to view similar software releases as competitors to one another. Yet, in an industry that prioritizes decentralization and the ability of multiple teams to work on a single platform simultaneously, it is inevitable that similar tools and solutions will emerge. Instead of trying different products against each other, both individual teams and the industry as whole stand to gain tremendously from looking for synergies and ways to leverage each other’s technology.
Far from being a competitor to IPFS or RAM, DISK could prove to be yet another vital component of EOSIO’s dynamic storage structure, with vRAM serving as a paging mechanism that allows developers to utilize all these tiers to create powerful, dynamic dApps.
Want to reduce your storage cost significantly? Get started with vRAM today!
Want to get real deep on how the vRAM System works? Read our technical article that clearly explains the inner workings of vRAM.
