Upgradeable Smart Contracts: Keep your data after smart contract upgrades
Iterable persistent storage in Solidity.
This article outlines a way to make smart contract upgrades and still keep all your data. Further it outlines a way keep full control over your data in case of necessary modifications.
Smart contracts are immutable by design. Once deployed a smart contract can not and should not be changed. This is a main concept of Ethereum.
This rule does not apply to the world outside of blockchains. Business requirements can change, security issues can be detected and people can simply make mistakes while developing smart contracts.
A way of bringing together the immutable world of blockchains and the ever changing world of businesses had to be found. The solution is a generic one and does not limit any future changes.
Eternal Storage to the rescue
In summary, the eternal storage pattern is a smart contract which only contains data an no business logic. It is the equivalent of a database in a web application. Because the eternal storage, as its name implies, is supposed to be used for the whole life cycle of a dApp it has to as flexible. This is achieved by a key-value store, which in database terms is called schema-less.
Here is the implementation of a key-value store for
uint256 in Solidity:
It consists of a mapping with a getter, a setter and a deleter. No more and no less. To access data a unique key is created. A wrapper function is used to not expose implementation details to the rest of the application. In the following example a balance is saved in a wrapper function:
Let’s compare this to saving a balance in a standard ERC-20 contract:
As shown above, without eternal storage the necessary code is much less and easier to read. So why would you want to use it?
Before the upgrade is after the upgrade
Sooner or later something will need to be changed in the code of the smart contract. No matter how small the change, a new version of the contract needs to deployed to a different contract address than before. By using the storage of the contract itself, like shown above in the standard ERC-20 contract, all data of the original version is not accessible anymore.
By using the eternal storage pattern, the data is kept in a separate contract. The deployment of a new version of the contract logic does not affect it in any way. So nothing is lost, nothing needs to be migrated and no time needs to be invested in restoring the old state.
There is a trade-off between increased code complexity and keeping data persistent. As the first one can be handled with a clean software architecture the trade-off is worth it.
The Only Thing That Is Constant Is Change
The eternal storage pattern provides great value to keep data persistent and accessible in a Ethereum blockchain. But it only solves the change of the logic of a dApp.
But what if the data itself has to be changed?
A concrete example of such a change is the change of the decimals of a ERC-20 token. The number of decimals specifies how divisible a token is. From 0 (not at all divisible) to 18 (divisible by 18 digits after the comma) and even greater if necessary. Ethereum itself has 18 decimals, so many ERC-20 tokens follow this standard.
But if a token is going to be bound to a fiat currency, 18 decimals do not make sense anymore. Two decimals are the standard for fiat currencies. To change the balances from a token from 18 to 2 decimals a simple division has to be done:
A division by 10 to the power of 16 calculates the correct representation of the balance when switching from 18 to 2 decimals. This needs to be done for every single account of our token.
A mapping in Solidity can not be iterated over, this is a limitation of the language itself. As the eternal storage pattern saves everything in a mapping it is impossible to loop through the accounts and execute the desired conversion.
Iterable mapping to the rescue
With the iterable mapping pattern it is possible to iterate over a mapping by storing all keys of it in an array. Whenever it is necessary to iterate over the mapping, the array is retrieved and the keys are used to access the values of the mapping. This comes with a higher gas cost for storage as an additional value is stored on the blockchain.
By using an array we can iterate over a single mapping. But an eternal storage contract uses many different mappings. As the mappings are grouped by type they will contain a lot of different values, like balances, allowances and decimals. If only one array by type would be used, it would be hard to keep track which key belongs to which variables of this type. The solution to this is to use one array by variable. For example: one array for balances, one for allowances and none for the decimals, as the value is a global one and does not need to be iterated over.
This solves the iteration problem of the eternal storage, but is not a generic solution. An eternal storage must be as generic as possible. It is not possible to know at the time of development which types of variables will need to be used during the whole life-cycle of a dApp. Arrays are not generic, but very specific. They need to be replaced with a generic solution.
Iterable mapping without arrays
The most generic data type that exists in Solidity are mappings. Therefore the ideal candidate to save the keys of a mapping is another mapping.
It sounds counter-intuitive at first, but mappings can be used to simulate an array. By this way the solution allows iteration over a mapping and stays generic at the same time.
A code example to store an account of a balance in another mapping looks like this:
The function retrieves the size of a list, inserts the value at the end of it and increases the size of the list by one. Whereas:
_listIdis a random unique value. This allows to add as many lists of keys as needed. So over time lists can be added or deleted as desired.
"KEY_SIZE”is a hard coded value which is used to access the size of the list. The size is needed to know at which index a value has to be inserted. When retrieving the list, the size is needed to know when the end of the list is reached.
"KEY_VALUES"is a hard coded value, which, together with the index, is used to store the desired key.
This function uses an eternal storage contract for all it data to store. Therefore all the benefits, like not being affected by upgrades of new logic contracts apply to it.
To retrieve the list with all its keys, the following function is used:
The function retrieves the size of a list, iterates from the first to the last index of it and saves the keys in an array. This array is returned at the end of the function.
The hard coded values
"KEY_VALUES" are the same as in the function to add the key.
for loop is used, to retrieve all keys in the example above. As the size of the keys increases, so will the time to retrieve them. This could lead to timeouts from the nodes, that are used to request the keys. A solution to this problem is to only retrieve parts of the list instead of everything at once. For the sake of brevity this is not implemented in the example above.
Coming back to the example of changing the decimals of the balances: The two functions above can be used to store and retrieve all accounts which have a balance. The accounts can be iterated over and the necessary conversion can be done.
We have published the fully working implementation of the iterable eternal storage contracts on GitHub.
What are the alternatives?
There are of course alternatives to this approach. We investigated them, but found that the Iterable Eternal Storage approach is the best match for our use case. If you want to know more about our decision process, please feel free to reach out in the comments.
Data can be kept after smart contract upgrades by using the eternal storage pattern. But the changes do not stop there. The data needs to be iterable to modify it whenever necessary. By applying the iterable mapping pattern without arrays, the two patterns can be merged and make the data of smart contracts as flexible as possible for whatever changes a dApp will need in the future.