Looking back at the Ethereum 1x workshop 26–28.01.2019 (part 3)

This is continuation of the part 1 and part2

Problems with large (and growing) state

Failing snapshot sync

Described in part 1

Duration of snapshot sync

Described in part 1

Slower block sealing

Described in part 2

To add to this, I have prepared a chart that shows number of SSTORE operations in a block, and number of those SSTORE operations that are “naked”, i.e. writing storage items that were not read or written before within the same block:

Moving average of SSTORE per block (blue), and “naked” SSTORE per block (black)

This should give an idea about the order of N2 in the formula shown in the part 2:

6*N2*log(Sdb) for storage item modifications

N2 is “naked” SSTOREs, which is currently on the order of 50 per block on average. That means we have 300 database accesses on average at the block sealing time, only due to SSTORE operations.

For non-storage state access, there are almost no naked writes, meaning that any update to the accounts is almost always preceded by the read.

For Turbo-Geth, there needs to be a different analysis, because unlike in other Ethereum implementations, reading from the state does get require warming up the trie cache. That means more work when sealing the block, in exchange for less work when reading the state. I might produce separate graphs for that case.

Slower processing of transactions reading from the state

Analysis of the reading of the state during the processing of transactions is similar to the analysis of the updates, and the formulae have the same form. The main difference is that updates can be cached during the block processing and executed at the end of the block, whereas reads have to be performed when requested, because further execution generally cannot proceed without the value being available. I tried to exemplify this crucial difference in these two diagrams (they only show “naked” reads and writes):

If we only do writes, all the interactions with the database can be delayed until the sealing
If we do reads (that miss the cache), the execution has to wait until the database interaction is done

Hopefully this shows that although state writes seem to have bigger performance impact than reads, it is not always that simple. Lots of state reads in transactions make them execute slower, generally without the ability to delay it to the end and execute all database interactions in one batch (as we can do with writes). How many state reads are happening in Ethereum blocks? Here are couple of charts on that. First one shows account reads (yellow) and “naked” account reads (black). That huge tower in the middle is from 2016 DoS attack, when tens of thousands of dust accounts were created in each block.

This chart shows the more recent period in more detail (tower removed):

Next chart is for SLOADs:

All in all, recent blocks do on average 200–300 “naked” account reads, and around 500 “naked” SLOADs. If we use the formulae from the part 2, we can say that, given the cold caches, it can translate to up to 4*300 + 6*500 = 4200 database reads per block on average.

With the current model of execution, all these reads need to happen sequentially. And that, combined with the sheer number of reads, makes the block processing times extremely sensitive to state read latencies. This could explain why HDDs are simply not performant enough to cope (all those reads are for random locations in the database). It also needs to be an important consideration when researching network attached storage for the state.

Again, Turbo-Geth requires a slightly different analysis, because it always generates 1 database read per state read, so in that case we would have 300+500 = 800 database reads per block on average. But it currently does not optimise database reads at the sealing stage, which is work in progress.

I will not list all the effects of slow transaction processing due to state reads, but there are more mitigations to explore.

Possible mitigations (in addition to the mitigations for the block sealing):

  1. Explore speculative concurrency of transaction execution. I have started another data analysis to see how much can be gained from that. To be published in one of the next parts.

Block gas limit increase and the State fees (formerly known as State rent) share initial steps

To appear in the part 4 or later

Stateless contract pattern is discouraged by the current gas schedule

To appear in the part 4 or later

eWASM interpreters could be a sensible first change even though gas cost might not be practical in the beginning

To appear in the part 4 or later

Chain pruning will become more relevant as we start constraining the state growth

To appear in the part 4 or later

Ethereum protocol changes do not need to take a year to be prepared

To appear in the part 4 or later