RFC: Telos Resource Improved Management Plan
Managing EOSIO Resources
Recent events on various EOSIO chains have illustrated operational weaknesses in the current design of how these chains manage their resources. In short, the combination of inexpensive CPU resources purchasable from the REX (Resource Exchange) staking pool makes it extremely inexpensive to flood an EOSIO network with usage. This usage can then overwhelm the ability to free up staked REX resources, locking up the system and preventing use for many. This is the situation that has been plaguing the EOS mainnet for weeks, driven first by an attacker who was able to drain 30,000 EOS from EOSPlay and later by a coin meant to consume all EOS free resources by incentivizing “resource mining” to earn tokens in a seemingly valueless project. There may be other potential network attacks using this exploit aside from theft and crippling congestion. Those entrusted to operate EOSIO networks must address this situation.
Telos is not exempt from this exploit. We have been spared the gambling dapp exploit because there are few casinos operating on Telos and those that exist have very limited volume, making such an attack economically non-viable. We have experienced a version of this worthless coin mining and have seen little impact in usage largely because Telos users had the benefit of already seeing the drastically falling value of these coins from the trading pattern on EOS — and therefore have declined to participate in meaningful numbers. However, Telos should not become complacent about these attacks. As more dapps deploy on the network and the Telos value rises, our network will become a more attractive target. In fact, the very high amount of REX staking on Telos makes such an attack much less expensive than on other EOSIO chains. Fortunately, Telos has a culture of facing problems head on and implementing solutions. The Telos community should act swiftly to address these areas of concern.
The TRIM Plan
It has been noted by prominent members of the EOSIO community that, in fact, this exploit is really just an outcome of the system operating as designed. While this is true, it points to a flawed design. When flaws are exposed, they must be addressed and the system improved. The TRIM Plan seeks to address these flaws and improve how resources are managed on Telos to keep the network usable for all and greatly reduce the incidence of attacks that render the network essentially useless for many users.
No single solution has emerged as a magic bullet to address this, despite much consideration by those knowledgeable about EOSIO software. This suggests that there is no simple solution possible. Rather than an exploit of a bug in the code which could be fixed and redeployed, this is an edge-case attack on the way that the system is designed to operate. Therefore, in improving how network resources are managed, it’s likely that the most effective solution will consist of several actions, each designed to mitigate some element of the attack. Perhaps none of these alone would be highly effective. This is just the nature of problems: some have elegant solutions, others just need to add more goalies to better protect the net.
The TRIM Plan incorporates seven points that seek to give the users and block producers more latitude to protect general usability without imposing limitations on users’ ability to employ their staked or purchased network resources any way they see fit. In seeking to improve usability, it’s essential that Telos does not allow any centralized entity to assert control over any user employing their resources as they choose.
The proposed steps of the TRIM Plan are:
1. To reduce ease of REX purchase attacks and prevent a REX lockup, set a maximum amount of resources any account can purchase from REX without being whitelisted by the block producers and update the REX algorithm purchase cap to be configurable, allowing block producers to manage it as needed.
2. To ensure a small number of free transactions to every Telos user, direct some of the TED Plan (Telos Economic Development Plan) REX contributions to purchase resources for a pool offering a configurable amount of CPU/NET for a limited number of free transactions per user each day or week
3. To allow higher frequency users the ability to have more transactions without managing resources, implement tokens from wallets or explorers that provide guaranteed transactions in exchange for inexpensive tokens.
4. To make attacks more expensive, revise the REX bancor algorithm to set a floor price for REX purchases configurable by block producers.
5. To allow users to not need any resources to interact with apps, encourage all apps to utilize the ONLY_BILL_FIRST_AUTHORIZER feature.
6. To acclimate users to high usage situations, transition Telos to low-leeway mode at all times.
7. To reduce access to the network from users seeking to exploit an app that individual API node operators or block producers deem harmful to the network, allow API node operators to filter access to such contracts and individual block producers to blacklist accounts.
None of these remedies threaten the decentralized nature of the Telos network and no central party controls their implementation.
Step by Step
Below is a breakdown of each action in the TRIM Plan.
1. To reduce ease of REX purchase attacks and prevent a REX lockup, set a maximum amount of resources any account can purchase from REX without being whitelisted by the block producers and update the REX algorithm purchase cap to be configurable, allowing block producers to manage it as needed.
In one version of this attack (the first demonstrated on EOS), a single account could buy any amount of REX resources up to the cap and singly overwhelm the system. As I noted in my previous article, Addressing the EOSIO REX Exploit, limiting the percentage of REX that any single account could purchase would make such an attack more onerous, but more importantly, it would encourage highly used apps to stake at least some percentage of the resources they need in CPU and NET.
An implementation of this modification has been written by Ryan Jones from the block producer Teleology and is awaiting testing on the Telos testnet. It is configurable by the block producers and allows them to whitelist certain accounts to allow them to purchase REX amounts over this general cap. Following testing, the code would need to be implemented by the block producers.
To increase the price of REX purchases as demand increases, REX uses a bancor algorithm to set its pricing. To prevent this price from approaching infinity in extreme edge cases, a cap is placed on the total amount of REX resources that can be purchased at any time. However, this limit was set unnecessarily low as a default. REX sales cease once they reach 80% of the available amount. The optimal level is unknown, but almost certainly higher than this, as the current situation on EOS demonstrates that REX can effectively be halted rather inexpensively at this level.
The simplest solution is to deploy new code to increase this level. Rather than hard-coding a new value, it is preferable to make this figure configurable by a 2/3+1 vote of block producers so that it can be rapidly updated in the future without requiring new code to be deployed.
To implement this, the Telos code must be revised and tested by the Telos Core Developers (TCD) and implemented by the block producers.
2. To ensure a small number of free transactions to every Telos user, direct some of the TED Plan REX contributions to purchase resources for a pool offering a configurable amount of CPU/NET for a limited number of free transactions per user each day or week
Telos makes it free to create new user accounts, but grants these new accounts with very low stakes of CPU and NET (0.9 and 0.1 TLOS respectively) which will likely be reclaimed in the future to allow more free accounts to be created. To ensure that every Telos account can always perform a minimum number of transactions each day or week, a pool could be created to facilitate this. While the TED Plan is in effect, this pool could be funded by directing some of the monthly distribution from the Exchange Reserve Fund to buying REX resources instead of simply adding to the REX rewards pool. (As the TED Plan winds down in some years, funds could be allocated towards a permanent staking pool for this purpose.) For REX stakers, the effect will be the same returns but for REX buyers, the cost of REX resources will rise as some is being used for this pool.
The goal of this purchase would be to ensure that every Telos account holder could perform some minimum number of transactions each day or week even without any resources staked or purchased. This amount would be configurable by block producers but 3–5 free transactions per day or 20 per week would be reasonable initial values.
Jesse Schulman of CalEOS has been working on code to implement this, however it appears that Greymass also recently developed an application that accomplishes this same end. To implement this feature the TCD will evaluate the various approaches and adopt or create code to be implemented by the block producers.
3. To allow higher frequency users the ability to have more transactions without managing resources, implement tokens from wallets or explorers that provide guaranteed transactions in exchange for inexpensive tokens.
Users who perform more daily or weekly transactions than the free pool permits would normally need to stake TLOS tokens for resources or purchase them from REX. As a middle ground for more frequent users, who may not be sophisticated about the ins and outs of EOSIO, or who simply choose not to manage resources, wallets and explorers could use tokens to ensure free resources. This allows even frequent users to have an alternative to staking which makes Telos more palatable for general computer users, aiding mass adoption.
Marlon Williams of Telos Miami has already instituted such a token with the SQRL token that allows users to consume SQRL in lieu of using their own resources. Any number of other wallets may use a similar token for these purposes, greatly aiding usability for higher volume users who do not wish to manage resources. This will also create a market for transactions separate from REX resource purchased which will drive further efficiencies within the system.
4. To make attacks more expensive, revise the REX bancor algorithm to set a floor price for REX purchases configurable by block producers.
The price of REX resource purchases is dictated by the bancor algorithm in the eosio.rex system contract. However, the bancor algorithm has proven itself to be less configurable than is optimal for the management of large decentralized networks. As described in this deep-dive into the bancor algorithm as used for REX, the primary controls on the algorithm are the “connector weight” operator (total_rent) and the amount of tokens injected into the contract through lending and borrowing. This low level of manageability is responsible for several shortcomings seen in EOSIO, such as the explosive price of EOS RAM at launch. Telos addressed this by modifying the initial amounts of RAM in the contract to prevent the rampant speculation seen on EOS, but the difficulty of sufficiently managing bancor-based contracts persists. This is particularly true of REX resource pricing because the amounts of resources are constantly in flux. Reconfiguring the connector weight control (total_rent) is one possibility and should be tested empirically. If it proves to be of little practical utility then a better algorithm is needed. It is possible to modify the bancor algorithm to include a floor price for resources. This will allow REX to be more expensive to purchase, thus increasing the cost of resource attacks.
Currently, the ratio of TLOS to REX for purchases is around 3,000:1. This means that 3,000 CPU worth of resources could be purchased from REX for one month for just 1 TLOS. This high ratio encourages attacks by making them extremely inexpensive. This is a by-product of the high staking rewards paid into Telos REX via the TED Plan. Other points in the TRIM Plan can be expected to increase this price to some extent.
Setting a floor price, configurable by block producers, of around 600:1 to 800:1 would mean that Telos REX resources would still be extremely inexpensive for legitimate purchasers, but more expensive for those seeking to exploit the network or lock up usage. There are two desirable by-products of such a modification: those staking resources to REX would receive a somewhat higher rate of return due to extra funds paid into the pool, and the TLOS price may appreciate as its true utility of staking to reserve resources becomes slightly more valuable. While mathematical modeling of these resource model adjustments is worthwhile, because there is little data to base decisions on, moderate adjustments accompanied by empirical observation and further adjustment is likely the most practical approach.
To implement this, we can test changing the bancor total_rent parameter using the setrex action. This should be tested on the Telos testnet before considering the update on the Telos mainnet. If this proves insufficient, the Telos code must be revised and tested by the TCD and implemented by the block producers.
5. To allow users to not need any resources to interact with apps, encourage all apps to utilize the ONLY_BILL_FIRST_AUTHORIZER feature.
The activation of EOSIO v.1.8 protocol features in October created a number of improvements in how Telos operates. Of these, the most important to this plan is ONLY_BILL_FIRST_AUTHORIZER, which allows apps to pay for any resource usage by those using their apps. (This is how SQRL tokens are used to pay for transactions, for example.) Encouraging the widespread usage of this feature by apps will remove the need for users to manage resources for most app interactions. This means that the previously mentioned changes of enabling the chain to pay for the first few transactions and special tokens to cover other usage, need only apply to transfers and other types of transactions not made with an app.
This change requires no new code at the system level, only informing Telos-based apps of how to utilize this feature. This is already occurring as new and existing apps are informed of how to apply these changes.
6. To acclimate users to high usage situations, transition Telos to low-leeway mode at all times.
Telos currently performs far fewer transactions than its capacity. While the network is in this state a high amount of leeway is given to any account that has already used its staked resources to execute transactions anyway. As a result, users may become accustomed to staking less TLOS for resources than they actually require. At times when there is much higher network usage, this leeway is removed or eliminated and accounts can only perform the transactions they have staked or paid for. This is currently the situation on EOS.
The lack of resources is compounded by the unfounded expectation of being able to perform a large number of transactions with few staked resources as when the network is under a low load and high leniency is given to users.
By removing or greatly lowering the leniency offered to users with insufficient staked resources, Telos users will become accustomed to more realistic expectations when the network is under a heavy load. In conjunction with the other actions recommended under this plan, reducing the resource usage leeway for accounts would encourage all users and apps to reserve an appropriate amount of resources for their needs. This also increases the overall amount of resources accounts are likely to stake, reducing the impact of resource type attacks. As an ancillary benefit, the TLOS price may appreciate as the token becomes more needed for its primary purpose of reserving network resources. With the other accommodations to users under this plan, casual users are unlikely to be effected by this and higher volume users will learn appropriate staking while the network is not under a high load, which will make adjustments far easier. This step would ideally be implemented after many of the prior steps to ensure smooth use of resources by casual users.
7. To reduce access to the network from users seeking to exploit an app that individual API node operators or block producers deem harmful to the network, allow API node operators to filter access to such contracts and individual block producers to blacklist accounts.
Most Telos users interact with the network though an API node operated by a block producer, app developer, or other party, rather than operating their own node on the network. While each Telos block producer is required to provide an active API node for accessing the system, it is not required that they permit every type of action via this API node. API node operators who are not block producers have absolutely no requirements as to what types of actions they may pass or block. This is essential in a freedom-based system. For example, a node operator may wish to support the network but not support interaction with a particular app that is against their beliefs, values or interests. No one can prevent an app from deploying on a truly public, decentralized network like Telos, but API node operators do not need to interact with them. Instead, they can block certain contracts on their own nodes. This may also be necessary for some node operators to comply with local laws.
An app that does not receive service from any API nodes is free to operate their own API node to allow users not operating their own nodes to interact with their app. Users who wish to interact with contracts filtered by other node operators may run their own node to do so. This is decentralization in action.
In the case of an app that is behaving in a way that any given API node operator deems harmful to the network, that operator is free to block interactions with that app from passing through their node. The app operator is free to inform users of API nodes that will process their transactions or to provide their own. So if a “resource mining” app threatens to overwhelm the network and make it unusable, API nodes may restrict access to it. Certainly, this does not eliminate access to that app, but it is likely to markedly restrict it to users who run their own network node or can access a limited number of public API nodes, thus reducing the impact on the network to more acceptable levels.
A more extreme version of API node filtering is block producer filtering of interactions with an app or contract. This would likely be considered a last resort by many block producers to be considered in the event of a malicious or unintended attack on the network’s availability to the majority of users. The Telos governance documents (specifically the regproducer agreement) prohibit eliminating or reordering transactions for profit, but do not prevent block producers from blocking transactions that they deem to be attacks on the network, illegal in their jurisdiction, or for some other reason not related to unfairly profiting from their privileged position. This is commonly referred to as “blacklisting” accounts or contracts. Accounts could be blacklisted for attacking the network, for engaging in illegal activity such as child pornography, because they hold suspected stolen funds or other reasons.
When a block producer blacklists and account, other block producers may still process these transactions when it is their time to produce blocks. These are individual decisions made by each block producers and if the Telos users feel that this is undesirable, they can vote to remove any block producer filtering transactions that they do not agree with. Moreover, if 2/3+1 of the block producers view this filtering as being against the terms of the regproducer agreement, they may sanction the block producer doing it. This is how a governed delegated proof of stake is designed to work.
Therefore, when network threats are particularly dangerous to the overall usability of the network, each Telos block producer may, based on their own judgement, elect to blacklist transactions to that contract. They will be answerable to the Telos voters and other block producers for their decisions. (Those block producers who allow such transactions to continue at the expense of network usability will also be judged by the users.) These decisions will be made by each individual block producer, not by a vote on chain. It should not be considered censorship by the chain, but merely the individual decision of each block producer. Only when most or all of the block producers opt to blacklist an account will it become unusable and the Telos voters will have the ability to vote such block producers out of service if this is against their wishes. Therefore, any app blacklisted by all the block producers so that they can no longer operate will have been blocked with the tacit approval of the Telos tokenholders.
Implementing the Plan
Unlike the TED Plan, the TRIM Plan deals with the operation of network code and functions which the Telos block producers are already empowered to revise. Therefore, no governance amendment would be required to implement these changes as long as 2/3+1 of the Telos block producers agree.
Additionally, no deep protocol changes are required to implement this plan. Each change made to the EOSIO code requires testing every time the code is updated in the future. This not only increases the expense of running the network, but also reduces the potential responsiveness of Telos Core Developers and block producers in the future should critical patches need to be deployed because these will require testing to incorporate such changes. For this reason, it’s preferable to keep the core Telos code as similar as possible to the EOSIO reference software.
Several aspects of this plan are currently underway. Others require no coding, only education and advocacy. Some will require new code to be written, tested, and implemented. Once fully implemented, Telos will have a variety of controls by which the block producers may rapidly respond to any future attacks on the network. Users will also have greatly simplified resource management options to the point where only power users and app developers need ever pay attention to resource management. These are two very large wins for usability and further network resiliency of the Telos blockchain network.
Support
The following Telos block producers and other parties support implementation of the TRIM Plan:
GoodBlock, Telos Miami, CalEOS, Infinitybloc, BlindBlocArt, EOSZA, Teleology, Cryptosuvi, Scatter, EOS Tribe, Telos DAC, Telos Voyager, Telos Global, Telos UK, Malta Block
