On Efficient Ethereum Storage

0age
Coinmonks
Published in
11 min readMar 4, 2019

--

As you make headway in the quest to reduce gas costs wherever possible, you inevitably discover that on-chain storage is frequently the most significant expense. There are many well-established ways to reduce these costs, such as verifying Merkle proofs against a root hash, storing IPFS hashes instead of placing the data directly on-chain, and packing multiple values into individual 32-byte words (perhaps by leveraging efficient addresses). Rather than go over each of these methods in detail, we’re going to investigate a new, bizarre technique.

Here’s some context: CREATE2 enables the creation of metamorphic contracts, or contracts that can be redeployed with new runtime code. The current prevailing mentality is that they are an abomination and should be avoided at all costs. The contrarian opinion is that metamorphic contracts are actually pretty cool, and can serve as flexible, lightweight upgradeable contracts. To add some fuel to the flamewar, consider this novel and slightly-unsettling application for metamorphic contracts: they can use their runtime code as a cheaper source of dynamic storage in a flagrant abuse of current gas mechanics.

Hodl on a sec there, pal: did you just say runtime code? As in the big, long 0x60806040… thingy? In place of storage? How does that even work?

In short: you start by storing the data the usual way at a factory contract, which then deploys a metamorphic contract that will retrieve the data and insert it into its runtime code, along with an initial control word that enables the factory to SELFDESTRUCT it. Then, you read the data from the metamorphic contract with EXTCODECOPY.

The better question, then, is why would anyone in their right mind take this circuitous route just to store and read data?

Well, SSTORE costs 20,000 gas to store each 32-byte word of data, while CREATE2 costs 32,000 gas (with a little extra for the cost of hashing the contract initialization code), plus 200 gas for each byte stored to the runtime code of the contract, which comes out to 6,400 gas for each stored word. We can write our data to storage temporarily, deploy a storage contract that pulls in the data and places it in its runtime code, and then delete the temporary storage to get a 15,000 gas rebate for each word. This gives us a total baseline cost of 11,400 gas per word, which is 8,600 gas cheaper than SSTORE.

From here on out, we’ll refer to this method of using the runtime code of a metamorphic contract for dynamic storage as RSTORE.

RSTORE: Save More (Sometimes)

So, as long as the size of the data we need to store can offset the added overhead of deploying a contract, there’s a case to be made for using contract runtime code instead of contract storage. In practice, there are a few extra steps and tricks, but here’s the gist of it: using RSTORE is cheaper to initially set up than SSTORE as long as more than one word needs to be stored. It eventually levels out at about two-thirds the cost of standard storage as you add more words into the mix.

Another observation to note: for dynamically-sized storage arrays like bytes and string, you don’t need to store the offset and length, since the “offset” is the metamorphic contract address and the length can be determined via EXTCODESIZE.

There’s Always a Trade-off

Updating storage via RSTORE is more expensive than with SSTORE, and serves as the Achilles’ heel of this method. Since it only takes 5,000 gas to update an existing storage slot, there’s no appreciable savings by utilizing temporary storage to populate the metamorphic contract’s runtime code. The cost of updating storage levels off as you add additional words until RSTORE is roughly twice as expensive as SSTORE.

We partially offset the additional cost via the 24,000 gas rebate when we SELFDESTRUCT the old metamorphic contract, but the length of the old runtime code will not impact the size of this rebate, and we still have to pay 5000 gas for the SELFDESTRUCT opcode itself. Furthermore, you have to update all of the storage at once, though you can of course break up your storage into multiple contracts if needed.

RLOAD: A Reading Rainbow 🌈

Here’s the kicker: the real cost savings don’t come from writing the data, but rather from reading it. Every word retrieved via SLOAD costs 200 gas, but if we retrieve the data from runtime code, we just pay a flat fee of 700 gas for EXTCODESIZE and 700 for EXTCODECOPY (or just EXTCODECOPY for fixed-size data) and a paltry 3 gas for each word. Using this method, which we’ll refer to as RLOAD, costs a smidge more for retrieving small amounts of data, but provides massive savings as the number of words increases: it only costs about one-third as much to retrieve large amounts of data.

Working Remote — No Pants Required

Finally, there’s even an application where RLOAD is always cheaper than SLOAD. When another contract needs to access the storage, they can just get the runtime code of the appropriate metamorphic storage contract directly, avoiding the cost of an additional CALL or STATICCALL along with the extra cost of processing the call on the receiving end. It’s less expensive for data retrieval of any size, and as much as one-fourth as expensive for retrieving large amounts of data, which makes it a fantastic fit for registries and other applications that need to make storage available to a wide variety of consuming contracts.

The Raw Reality of RSTORE & RLOAD

To consider the relative cost of RSTORE and RLOAD vs. SSTORE and SLOAD, we can calculate a cost multiplier based on the size of the data being stored and retrieved (removing 21,000 gas for the transaction overhead) and comparing the ratios of each. Bear in mind that these multipliers can change based on how much of the rebate you’re eligible for, as rebates are capped at half the total cost of the transaction.

Basically, you’d want to consider using RSTORE and RLOAD if you need a read-heavy source of storage that doesn’t change too often. Moreover, if storage doesn’t need to change, you can do away with metamorphic contracts entirely and just deploy static contracts with runtime code that contains the necessary data — a great example of where this could come in handy is for publishing lengthy proofs that can then be cheaply verified on-chain.

To put RSTORE & RLOAD in more concrete terms: let’s say you have a permissioned token, and that it requires that four attributes are set in a registry for a given address in order for that address to receive tokens. The attributes probably won’t be modified often, but do still need to be modifiable. RSTORE uses ~30,000 less gas in order to initially set all these attributes, and RLOAD saves ~2,000 gas every time they’re checked during a transfer. If you have 1,000 users a day setting up attributes and 5,000 token transfers a day (which might make it one of the most widely-adopted tokens right now 🤣), that adds up to 40 million gas saved every day. At the current standard gas prices (5 gwei) and price of Ether ($134), that translates to almost $10,000 in annual savings.

Crafting a Metamorphic Storage Contract

As you might expect, deploying a metamorphic storage contract with fixed initialization code and near-arbitrary runtime code involves some twisted witchcraft that Solidity isn’t really equipped to handle. You know what is equipped to handle it? Raw EVM opcodes. We’re taking Roy off the grid.

Metamorphic Storage Contract Initialization Code (69 bytes)0x5860008158601c335a630c85c0028752fa153d602090039150607381533360601b600152653318585733ff60d01b601552602080808403918260d81b601b52602001903ef3

To start, we need to retrieve the code to store. The metamorphic contract expects the creating contract to make a getTransientStorage() view function available that will return this code, which has a function selector of 0x0c85c002. The first 17 bytes of the initialization code is devoted to just that.

pc op operation_name   [stack_items]                 <memory>00 58 PC               [0]
01 60 PUSH1 0x00 [0, 0]
03 81 DUP2 [0, 0, 0]
04 58 PC [0, 0, 0, 4]
05 60 PUSH1 0x1c [0, 0, 0, 4, 28]
07 33 CALLER [0, 0, 0, 4, 28, caller]
08 5A GAS [0, 0, 0, 4, 28, caller, gas]
09 63 PUSH4 0x0c85c002 [0, 0, 0, 4, 28, caller, gas, selector]
0E 87 DUP8 [0, 0, 0, 4, 28, caller, gas, selector, 0]
0F 52 MSTORE [0, 0, 0, 4, 28, caller, gas] <selector>
10 FA STATICCALL [0, success => 1]

Now that the data resides in the return buffer, we can proceed to set up the bottom of the stack to prepare the final runtime code placement. Our getTransientStorage() function returns an ABI-encoded bytes array, which includes two extra words at the front for the offset and length, respectively. Our runtime code will have only one extra word at the front — a control word that allows the metamorphic contract’s creator, and only the creator, to SELFDESTRUCT the contract via a simple, empty CALL to it. So, we’ll subtract 32 from whatever length we get back from RETURNDATASIZE and sock that away for later. (As a quick aside, this means that we can actually save partial words in runtime storage and only pay for what we use, unlike regular storage where every slot uses the full 32 bytes.)

pc    op    operation_name    [stack_items]11    15    ISZERO            [0, 0]
12 3D RETURNDATASIZE [0, 0, size]
13 60 PUSH1 0x20 [0, 0, size, 32]
15 90 SWAP1 [0, 0, 32, size]
16 03 SUB [0, 0, size - 32]
17 91 SWAP2 [size - 32, 0, 0]
18 50 POP [size - 32, 0]

Next, we’ll begin to construct the control word and place it into memory in four parts, one segment at a time. The first three parts (or 27 bytes) of the control word are based on the child contracts created by the GST2 variant of GasToken and are as follows:

op operation_name73 PUSH20 *factory_address*        // Hard-code the factory address
33 CALLER // Get the caller's address
18 XOR // Equals 0 iff address is caller
58 PC // no jumpdest to this PC, so:
57 JUMPI // revert if caller != address
33 CALLER // send any funds to the caller
ff SELFDESTRUCT // selfdestruct the contract!

Without further ado, we proceed to place them in memory.

op operation_name       [stack_items]  <memory>60 PUSH1 0x73           [size - 32, 0, *push20*]
81 DUP2 [size - 32, 0, *push20*, 0]
53 MSTORE8 [size - 32, 0] <*push20*>
33 CALLER [size - 32, 0, caller]
60 PUSH1 0x60 [size - 32, 0, caller, 12_byte_offset]
1B SHL [size - 32, 0, caller_with_12_byte_offset]
60 PUSH1 0x01 [size - 32, 0, caller_with_offset, 1]
52 MSTORE [size - 32, 0] <*push20_caller*>
65 PUSH6 0x3318585733ff [size - 32, 0, opcodes]
60 PUSH1 0xd0 [size - 32, 0, opcodes, 26_byte_offset]
1B SHL [size - 32, 0, opcodes_with_26_byte_offset]
60 PUSH1 0x15 [size - 32, 0, opcodes_with_offset, 21]
52 MSTORE [size - 32, 0] <*push20_caller_opcodes*>

We’ll set up the fourth part of the control word, which encodes the length of the data as a convenience, in tandem with the stack arrangement for RETURNDATACOPY to save some operations, since they use some of the same values.

op operation   [stack_items]                     <memory>60 PUSH1 0x20  [size - 32, 0, 32]
80 DUP1 [size - 32, 0, 32, 32]
80 DUP1 [size - 32, 0, 32, 32, 32]
84 DUP5 [size - 32, 0, 32, 32, 32, size - 32]
03 SUB [size - 32, 0, 32, 32, size - 64]
91 SWAP2 [size - 32, 0, size - 64, 32, 32]
82 DUP3 [size - 32, 0, size - 64, 32, 32, size - 64]
60 PUSH1 0xd8 [size - 32, 0, size - 64, 32, 32, size - 64, offset]
1B SHL [size - 32, 0, size - 64, 32, 32, size-64_offset]
60 PUSH1 0x1b [size - 32, 0, size - 64, 32, 32, size-64_offset, 27]
52 MSTORE [size - 32, 0, size - 64, 32, 32] <*control_word*>

Finally, we copy the data from the return buffer using RETURNDATACOPY, skipping the first 64 bytes with the offset and length, and placing them after the first 32 bytes occupied by the control word, then return to deploy the metamorphic contract with all the data stored in runtime code!

op operation_name       [stack_items] <memory> {deploy_runtime_code}60 PUSH1 0x20           [size - 32, 0, size - 64, 32, 32, 32]
01 ADD [size - 32, 0, size - 64, 32, 64]
90 SWAP1 [size - 32, 0, size - 64, 64, 32]
3E RETURNDATACOPY [size - 32, 0] <*control_word+data*>
F3 RETURN {*control_word+data*}

There are certainly ways to optimize this further and cover some sharp edge cases, but it gets the job done without too much fuss. Let me know if you have suggestions for optimization or other improvement.

A Wrinkle in Transaction Time

If you’re familiar with the mechanics of SELFDESTRUCT, you know that the contract is not actually removed from the state when the opcode is reached. Instead, it is scheduled for deletion in the transaction substate, which is only triggered at the end of the transaction. This poses a challenge for RSTORE, as it will require two separate transactions in order to update a given metamorphic contract’s runtime code.

However, we can work around this limitation by utilizing both a primary and a secondary storage contract. Then, we can update one and simultaneously SELFDESTRUCT the other, and use EXTCODEHASH to determine which contract to read from. This method allows us to update runtime storage in one transaction, but still only allows one update per transaction, and can result in stale inter-transaction data. Therefore, it’s advisable to use RSTORE in a single, atomic transaction.

Additional Homework for Extra-Debit

RSTORE and RLOAD can be further optimized based on your specific use-case by using an efficient deployer address and tighter packing for the control word and runtime data in storage contracts, by removing the secondary storage contract when not needed, and by using standard storage in place of temporary storage when more frequent updates are required. Regardless, this implementation demonstrates that it’s feasible to use metamorphic contract runtime code in place of storage to reduce gas costs, especially for read-heavy applications that don’t require frequent updates.

If you’d like to dig into the internals of RSTORE and RLOAD or have ideas for how to improve the technique further, take a look at the Github repo. If you think this could be useful to you and would like to try to implement it, be sure to have a plan in place should future changes to gas calculations (I’m looking at you, state rent) end up tipping the scales against this strategy. And if you still think metamorphic contracts are an abomination with no legitimate use-case, let me know so that we can get into an argument some time.

Thanks to Stephane Gosselin, Matt Czernik, and Santiago Palladino for the productive discussions around this idea and metamorphic contracts in general, and thank you for taking an interest in this odd, unconventional storage exploit.

Get Best Software Deals Directly In Your Inbox

--

--

0age
Coinmonks

Head of Protocol Development @ OpenSea (views are my own)