EOS Chain History Challenge: A Thing of the Past
While Block Producers strive to differentiate themselves from the competition in order to earn the necessary amount of votes that would place them in the top 21 BP list, they are also incentivized to cooperate out of an obligation to keep the ecosystem functional and productive. Providing dApps with a reliable source of chain history data is one such example of BPs cooperating in a meaningful way for the common good of the EOS ecosystem. As the EOS network continues to grow at a rapid pace, a cooperative approach to solving the history issue becomes crucial to the long-term prospects of the network.
What is a full-history node?
A full-history node is actually a misleading name. While every synced history node does contain the entire record of transactions of the EOS mainnet, the nodes are not the actual source of the historical documentation. We would still be able to retrieve each and every transaction ever made on the blockchain without full-history nodes, making the full-history nodes less crucial to EOS core functionality than people assume them to be.
What full-history nodes do provide is a layer of analysis on top of the historical data that makes it easier for dApps to process and use. In order to handle this type of data processing, the nodes need to run full-history plugins and solutions, which require a significant amount of RAM and high maintenance costs. dApp developers could use full-history nodes to track and query data for specific user accounts (e.g. pulling all actions or transactions linked to that specific account) but that type of full-history node analysis would require almost 2TB of additional data (and the associated RAM amounts).
Danger: Failure Ahead
Currently, only five BPs — Greymass, EOS Tribe, Cryptolions, EOS Canada and EOS Sweden — maintain the necessary hardware to store and index the entire ~2TB worth of full-history node. Out of the 2TB, roughly 210GB is used to store the entire chain data while the rest contains the information that is required to maintain their history plugin.
As previously reported by EOS42, during mid-November 2018, the exponential growth of the EOS blockchain led to a momentary crisis as all full-history nodes went down. With no publicly accessible endpoint, they could use to query the state of the blockchain, many applications on EOS were unable to process their users’ requests until a history node went back online. As time goes by and the size of the blockchain increases, maintaining a full-history node will become prohibitively more expensive and we could encounter something similar to what happened last November. Thankfully block producers and developers have prioritized long-term solutions to the chain history issue (for a look at some of the proposed solutions, and a short explanation about the November hiccup, read this excellent piece by EOS42, who contributed their expertise to this article).
100% of the Costs for 20% of the Data
Full-history nodes cost around $30k to construct, a rather expensive undertaking, especially for BPs outside of the top 21 who are not receiving active BP rewards. Adding maintenance costs and those associated with upgrading hardware as the blockchain grows contributes to the unsustainability of maintaining a full-history node. Full-history nodes also operate under the Pareto principle: 80% of the API request from dApps are seeking data relating to the most recent 20% of chain history, making full-history nodes incredibly inefficient. As the EOS blockchain evolves, so will the price of a full-history node. If a single BP were to decide to stop supporting a full-history node, the additional pressure that would place on the remaining nodes could severely constrain network performance.
Full-History-As-A-Service
Currently, the only revenue generated by BPs to offset the cost of maintaining a full-history node is the block rewards they earn. BPs maintaining full-history nodes depend on the market price of EOS and securing a spot in the top 21 in order to survive. The DAPP Network could assist BPs by giving additional incentives to keep more full-history nodes running by creating a history node Service Package.
DAPP Service Providers (DSPs) could provide full-history of the blockchain as one of their services, allowing developers to access the data in exchange for staking DAPP tokens towards their service package. They could offer different service packages according to the amount of history required by the dApp. For example, one package can be limited to offer the most recent 1,000 actions per account, while another package could price each query according to how long it goes back in history. The DSPs will have full autonomy over the types and prices of packages they offer.
This creates a stake for play model in accessing history nodes — developers only stake for DSP packages when they need to utilize the history nodes. DSPs are rewarded through DAPP token inflation, proportional to the number of tokens staked to their services, creating a potential new revenue stream to help cover the cost of maintaining a full-history node.
From Challenge to Opportunity
As EOS evolves beyond rudimentary use-cases and begins to power mission-critical applications, it becomes increasingly imperative to ensure that these dApps have a reliable source of historical data. We believe that the DAPP Network, as a decentralized network of DAPP Service Providers, can assist in creating the model Dan Larimer envisioned when saying “Market will set price for supply/demand of full-history nodes”.
