Validators or value-takers?

22 min readOct 31, 2022

Diving into the pools and dark forests of PoS Ethereum

“The panda will never fulfill his destiny, nor you yours, until you let go of the illusion of control.” — Master Oogway

Photo shows galaxy with screens reflecting reporting graphics and analysis

Introduction

It is not often that fate provides us blockchain analysts with an event as pivotal and rich in data as the Ethereum merge. For this reason, we wasted no time merging (pun intended) minds from Metrika and Miga Labs to assemble a crack team of analysts and engineers ready to delve into this fount of data. Our intentions were simple: collect, analyze, visualize, and educate.

The merge was a much anticipated moment for the blockchain community. All watched as Ethereum completed the first ever transition from a proof-of-work (PoW) consensus mechanism to proof-of-stake (PoS). The first stage in this evolution was the launch of the PoS Beacon chain in 2020; this ran in parallel to the PoW chain, miners continued to secure the PoW chain, whilst validators secured the Beacon chain. The two chains merged in September 2022, as the miners handed over the reins of consensus to the validators.

Ethereum nodes now consist of an execution layer (EL) which processes transactions, and was formerly the PoW chain, and a consensus layer (CL) which manages the PoS consensus. This means that we were able to complete pre- and post-merge analysis of the EL transactions, the CL validators, and both consensus mechanisms.

We were keen to gain an understanding of the general network operation since the merge and assert the anticipated effects the merge has had on a number of key blockchain metrics. We also made sure to sharpen our metaphorical axes in preparation for cutting into the dark forest of MEV.

In order to provide a holistic view of this unique event this blog addresses the following five questions:

Did the merge affect the operations of the CL?
Did the merge affect Ethereum’s decentralization?
How did the merge affect CL rewards?
How did the merge affect the EL?
How can we identify MEV blocks and what differentiates them from normal blocks?

Data sources & collection

The first step in our journey was gathering the data. We assessed the sources available to us and settled on the following:

Geth node’s JSON-RPC API — all execution layer data
Prysm node’s JSON-RPC API — EL block’s slot & proposer data
Lighthouse node’s JSON-RPC API — all consensus layer data
MEV relay’s data API — additional post-merge MEV data

We wrote Python parsers, using Pydantic, for the Geth and Prysm JSON-RPC APIs and both MEV APIs; these are available on our GitHub. We stored the data in our Elasticsearch cluster. For the consensus layer analysis, we developed a series of Jupyter notebooks and Python scripts to manipulate the data from our PostgreSQL database; they are all available on our GitHub.

It was agreed that we would collect a month’s worth of data pre-merge, 192,000 blocks, and another month’s worth of data post-merge, 7,000 epochs or 224,000 blocks (Block time decreased from ~14 seconds to 12 seconds). With all of this agreed upon and in place, we began collecting data and divided the analysis between the CL and the EL.

Consensus layer analysis

Whilst the execution layer manages transactions on Ethereum and implements the Ethereum Virtual Machine (EVM), the consensus layer’s role is, as the name suggests, to enable the chain to reach consensus and finality. It incentivizes validators by offering them rewards for conducting their duties, let’s get into this in more detail.

Rewards on the consensus layer

The consensus layer is formed by validators who perform tasks each epoch. These tasks include attesting to the validity of blocks, proposing new blocks, and/or maintaining valid block headers. As a result of performing these tasks, validators earn three types of rewards:

Attestation rewards
Proposal rewards
Sync-committee rewards

Let’s dig into each of these and see both: how they are performed and the value of the reward received.

Attestation rewards

In each epoch, a committee of validators is allocated to each of the epoch’s 32 slots, they then participate in consensus by voting on the correctness of the block proposed for their allocated slot. They are also required to vote on pairs of checkpoint blocks to enable the network to finalize. By performing these tasks, validators earn attestation rewards.

Attestations are processed and evaluated for each slot and rewards are applied in the following epoch. A validator’s attestation can be included up to 32 slots after the slot that they are attesting to. In other words, the first block in an epoch could have attestations included in the first block of the following epoch. Compared to other rewards, the attestation reward is the lowest one, but is also the most common, as they can be earned for every epoch that the validator is active.

As there are many validators allocated to each slot, they are randomly sorted into beacon committees. Every validator shares its vote with its beacon committee each epoch and then all of the committees votes are aggregated into a single Attestation. If some validators send their vote later, another attestation is created with these votes. An attestation contains (among others):

Source: hash of the last justified checkpoint when the attested block was proposed
Target: hash of the first block at the epoch the attested block was proposed
Head: hash of the attested block (also known as Beacon block root)
Aggregation_bits: An array of bits representing the group of validators from the beacon committee (whether they voted or not).

Proposal rewards

For each epoch, 32 validators are chosen, at random, to be block proposers. Each of these is in charge of proposing a block in one of the epoch’s slots. As there are more than 400K validators at the time of writing, the chance of being selected to propose a block is low, about 1 block every 62 days. This probability only decreases as the number of validators increases. To make up for this, proposal rewards are significantly higher than attestation rewards. Proposers earn rewards by including attestations and sync aggregate in their block.

Sync-committee rewards

Since the Altair hard fork, the Ethereum network has introduced the concept of sync-committees; these are a group of 512 randomly selected validators who sign block headers each slot. These headers are used by light clients so that they can trust which blocks have been validated without the need to fully download and process the Beacon chain.

During the 256 epochs (8,192 slots) that a validator is participating in a sync-committee, it receives an extra reward if it correctly participates in each slot: the sync-committee reward. Like the proposal reward, it is added at the state transition of every slot, not at the epoch processing. After 256 epochs, the list of validators in the sync-committee is refreshed. To check how much a validator, who successfully participates in the sync-committee, gains per slot, see here. The sync-committee reward, per slot, is not as high as the proposal reward, but it is earned across all 8,192 slots that the validator is participating in the committee.

Global analysis

We first looked at the behavior of the overall CL network, before grouping the validator set by node operator and segmenting by other classifications. Missed blocks and attestations are important metrics that provide an indicator of the health of the network. When comparing the number of missed blocks for 7K epochs pre-merge to 7K epochs post-merge, we observe a reduction of 36.40% in missed blocks. This might be related to the fact that missing a block post-merge has a double penalty of no CL nor EL rewards; thus one could expect validators to make every effort to avoid missing any block proposals. On the other hand, we do observe a slight increase in missed attestation flags, from 0.5%, 0.5%, and 2.9% pre-merge to 0.8%, 0.8%, and 3.3% post-merge for source, target, and head respectively.

As far as the number of validators is concerned, we observed a continuous linear increase of 3.8% from the beginning of the pre-merge period (428K validators) until the end of the post-merge period (444K validators). We also analyzed the frequency at which individual validators propose blocks. Overall, there are no noteworthy differences: during both periods, 90% of validators proposed either one or no blocks; however, some lucky pri̶c̶k̶s̶oposers proposed 6 or 7 blocks in the span of only 31 days, while most validators will propose only 6 blocks per year!

Pools

Decentralization

Next, we analyzed the level of decentralization pre- and post-merge. For that, we looked into the block proposal distribution for the 10 largest entities: mining pools pre-merge (determined by fee recipient and using Etherscan), and exchanges/staking pools post-merge.

Overall, we observed a rather similar decentralization level. For instance, pre-merge we see that Ethermine, F2Pool, and Hiveon Pool proposed 28.5%, 14.7%, and 10.2% of the blocks respectively. Post-merge, Lido, Coinbase, and Kraken proposed 29.9%, 14.2%, and 8.4% of the blocks. Nonetheless, it is important to keep in mind that Lido consists of 28 operators that are somewhat independent. This adds another level of decentralization that was not present in the PoW world. In addition, we observe notably more small-scale/solo staking (“Other”) participation (~36.0%) in comparison to small-scale/solo mining (~20.9%) likely due to the reduced infrastructure and energy requirements.

Lido operators are completely free to decide which client they use and what setup they run. Lido is governed by the Lido DAO and provides guidelines, such as increasing client diversity and using risk-management techniques; and a policy, such as how operators are expected to use MEV-Boost.

Rewards

To analyze the CL rewards we grouped validators into 42 of the most substantial entities, including solo stakers in the “Others” category, and plotted the total CL rewards on a logarithmic scale for each entity. About two-thirds of the entities observed a slight decrease of 4.4% in CL rewards from pre- to post-merge, probably related to the 3.8% increase in number of validators, decreasing the chances of getting important duties such as block proposals and sync-committees. It could also be partially linked to the increase in missed attestation flags mentioned previously.

The list of validators for most large pools can be obtained by inspecting blocks’ graffiti. One can also track the deposit addresses of the validators in order to get an idea of which pool they might belong to.

We also compared CL rewards to EL rewards for 7K epochs after the merge on a logarithmic figure; we found that CL rewards account for an average of 79.8% of the total rewards received by entities. Entities do show a significant variation in this regard; for some entities, the CL contributed 94.0% of their rewards during the period, while for others, just 67.6%. We found that small staking pools receive a greater proportion of their rewards from the CL than the EL block rewards; this is because small pools, with fewer validators, will propose fewer blocks in any given period.

The observed proportions meet expectations; on average a validator proposes a block every 61 days with 0.06ETH in EL rewards while in the same time period they will have earned about 0.20ETH in CL rewards from attesting. It is, therefore, no surprise that CL rewards are expected to account for 76.9% of a validator’s income.

CL Clients

Given that four of the CL clients have public pools with the same number of validators, we took this opportunity to check if there are significant differences in terms of CL rewards. The largest difference is only 4.6% and the mean difference is 1.7%, showing that client choice shouldn’t affect CL rewards. We did not perform a direct comparison with Lodestar as they have a different number of validators; however, when normalizing by validator count we also observe minimal differences.

Consecutive blocks

Lastly, we analyzed the number of consecutive blocks a pool has proposed in the months pre- and post-merge. The behavior is almost identical in both cases. We found chains as long as 10 consecutive blocks proposed by Lido validators and chains of up to 6 consecutive blocks proposed by Coinbase and Kraken validators. Consecutive blocks can provide a node operator with an advantage, known as multi-block MEV, as has already been documented. We also found a chain of 4 consecutive missed blocks in the pre-merge period, while in the post-chain period the maximum number of consecutive missed blocks was only 2.

Execution layer analysis

What we found throughout most of the EL data was that not too much had changed. This is a testament to the thorough testing of the protocol and clients, but isn’t particularly interesting to put in a blog post, so you’ll have to take our word for it.

Something that did change, however, was the block interval time! Post-merge, PoS, Ethereum no longer relies on a cryptographic race between miners, the block times are now deterministic and each slot is 12 seconds apart, whereas they previously ranged anywhere from about 5 seconds to about 25 seconds. To fully appreciate this pleasing determinism we plotted the block interval time.

Other than a few empty slots, it was clear that the rumors were true and the block time is now a constant 12 seconds. We also saw a slight increase in transactions per second (TPS), from around 12.1 pre-merge to 13 post-merge, that is a 7% increase! More on transaction count and TPS later.

MEV

Does a panda sh*t in the Dark Forest?

Maximum Extractable Value, or MEV, is as controversial as it is fascinating and has been under the spotlight ever since the merge. Its negative externalities were first well documented in the infamous “Ethereum is a Dark Forest” blog post, and it has since been the subject of extensive research.

MEV refers to the maximum value that can be extracted when producing a block, by including, excluding, and changing the order of transactions in a block. It first grew in popularity pre-merge, but now that there is no longer a fixed 2ETH block reward for proposing a block post-merge it is even more relevant.

The competition to get MEV opportunities included in blocks has caused gas fees to be driven up in the past and some forms of MEV can cause slippage of user’s trades. The positive effect of MEV is that it can fix economic inefficiencies through arbitrage and quick liquidations.

In practice, in pre-merge Ethereum, bot operators (searchers) sought to extract MEV by either paying high fees to increase the likelihood that their transactions were mined, or by fine-tuning their gas price choices in order to “time” their transactions right.

Flashbots optimized MEV extraction, in pre-merge Ethereum, by allowing searchers to submit bundles of ordered transactions, effectively transferring part of the ordering privileges from the miners to the searchers. These bundles were included by the miners provided they get more out of them than by mining regular transactions. While searchers can still set a gas fee, the preferred way to pay the miners for inclusion is via a direct transfer included in the bundle.

Flashbots’ mev-boost

MEV-boost Flashbot’s open-source middleware implementation of proposer builder separation (PBS) that is run by Ethereum validators and provide them access to a block-building market. It allows a validator to increase their staking rewards by selling their block space to an open market.

The mev-boost sidecar receives blocks from relays, which transfer and validate (supposedly) blocks from builders, which produce blocks from transactions provided by searchers, the mempool, and private channels.

Searchers

Searchers identify MEV opportunities from transactions within the mempool and create a “bundle” (collection of transactions) that extracts this opportunity. They then send this bundle to a Builder.

Builder

A builder collects together bundles from different searchers, public transactions from the mempool, and transactions from private channels, and produces a block with these transactions. It then sends this block to the relays.

Relay

A relay is an intermediary between builders and proposers (validators). It verifies the validity of the blocks provided by builders, selects the block with the most valuable payload, and then sends a header to the proposer of a given slot. A validator can connect to multiple relays through mev-boost, it registers with each relay every epoch. The validator’s mev-boost software receives block headers from each of the relays it is connected to and selects the one with the most valuable payload.

A proposer’s EL will have also produced a block, the CL then proposes the block with the most valuable payload between its execution layer’s block and the mev-boost block.

The reason that only a block header is sent to the proposer is so that the proposer cannot steal the MEV from the builder. The relay is trusted by the builder to keep the block transactions secret and is trusted by the validator to verify the block is valid and the reported rewards are correct.

Identifying MEV

“So this is all well and good, but what has the effect of the merge been on this mystical MEV?!” I hear you ask. Well, to inspect these effects, we must first be able to identify these MEV blocks, and sadly they can be quite good at flying under the radar.

Pre-merge MEV identification

By default Ethereum clients order transactions by gas price, selecting the transactions that will return the highest fee first. This isn’t always the case due to private transaction channels and clients allowing other configurations, but it is largely true.

However, the most notable exception to this rule is, you guessed it, MEV. Miners that were accepting MEV bundles would generally place these bundles at the start of their blocks regardless of the gas fee as they were incentivized by the direct transfers within the bundles. Therefore, a fairly accurate estimate for identifying MEV blocks is looking at blocks where transactions aren’t ordered by gas fee. Some false negatives may arise from MEV opportunities being exploited by mempool bots that get their transactions included by tuning the gas fee, however, this was deemed to be an acceptable exception as the miners are not necessarily knowingly including this MEV, so is it even miner extractable value at this point?…

MEV originally stood for Miner Extractable Value, however, this was updated to Maximal Extractable Value to reflect the fact that miners would no longer be a part of the Ethereum protocol after the merge.

An alternative method could have been to use the Flashbots mev-blocks API but this only returns Flashbots transactions which are a subset of the gas price ordering method.

Post-merge MEV identification

The MEV identification via transaction ordering method, mentioned above, is still applicable post-merge. However, now that MEV has been standardized by Flashbots’ mev-boost, methods for identifying MEV blocks are somewhat better defined. Each of the relays provide a JSON RPC endpoint that provides data on the blocks that they have relayed. The relays that we used are: Flashbots, BloXroute Max Profit, BloXroute Ethical, BloXroute Regulated, Blocknative, Manifold and Eden.

In addition, most block builders set themselves a block’s fee recipient and then make a transfer of the MEV value (less their cut) to the proposer’s fee recipient address as the last transaction in the block. This has been acknowledged by @metachris of Flashbots.

Manifold and Blocknative, however, set the proposer as the fee recipient and cannot be identified by using the last transfer of MEV value, so their built blocks were fetched via their JSON RPC endpoint.

Lastly, block builders also often put their tag in the block’s extra data field; this could be used to identify MEV blocks but has some fairly obvious limitations.

Comparison

We used three of the methods mentioned above to identify MEV blocks: transaction ordering by gas price, the final transaction in the block being from the block’s fee recipient, and the relay APIs. We then aggregated these by hour to get a proportion of MEV blocks per hour.

As one can see, pre-merge ~80% of transactions within a block were not ordered by gas price; post-merge this halved to ~40%, but then slowly crept up towards 60%. The other post-merge identification methods are very closely correlated to each other, showing an increase in MEV usage from ~20% to ~50% since the merge. The most likely explanation for the post-merge difference between the transaction ordering method and the other two is the “noise” of the transaction ordering method; in other words, The false positives caused by private transaction channels or non-standard client implementations. It does seem to be converging slightly with the other two methods, this may be an artifact of the mempool MEV bots starting to use mev-boost builders for the inclusion of their bundles so a larger proportion of the total MEV is captured by the last transaction and relay identification methods.

MEV adoption

Once we had determined how best to identify MEV we wanted to investigate its prevalence and adoption by node operators. We gave each node operator one of three classifications: has never used MEV; has always used MEV; has sometimes used MEV. In practice, a node operator’s validator may not always have the option to use MEV as there may be no MEV opportunities available or, post-merge, mev-boost may not receive any preferable blocks.

Using only gas price ordering, so as to be able to compare pre- and post-merge — and counting unique fee recipients, as a proxy for identifying node operators — we see most proposers, both pre- and post-merge, using MEV some of the time, with none using MEV all the time. Interestingly, the proportion of proposers who have never used MEV was significantly larger post-merge. We speculate that this is due to validators delaying their use of mev-boost post-merge, possibly to observe its performance on Mainnet. It’s important to remember that the number of unique validators is larger than unique miners (yay decentralization!).

By only using gas price ordering to detect MEV:

MEV effects

Since the merge, Flashbots’ mev-boost seems to be acting as a metaphorical flash-light, illuminating the previously “Dark Forest” into more of a dim forest. Taking advantage of this, we began comparing MEV blocks to non-MEV blocks, both pre- and post-merge, around several interesting metrics: transaction count, gas used, block rewards, and transaction fees.

Transaction Count

We began by counting the number of transactions in each of the blocks and plotting this in a histogram, separating MEV blocks from non-MEV blocks. Blocks without any transactions account for 3.88% pre-merge and only 0.50% post-merge. We have excluded those blocks containing no transactions as these can likely be attributed to client bugs.

We are only showing up to 800 transactions per block, although we have seen some blocks with over 1400 transactions both pre- and post-merge.

Pre-merge it is clear that the MEV blocks generally have more transactions in them, and both MEV and non MEV have very similar ranges.

Potential explanations include the fact that MEV bundles include additional transactions for the extraction of value from mempool transactions, and that when there are few transactions that meet the network’s base fee requirements it is less likely that there will be any MEV available, resulting in low transaction count, non-MEV, blocks.

The difference is also noticeable post-merge and there are regularly more transactions in an MEV block, especially when looking at the number of blocks with >200 transactions. This may indicate that the specialized MEV block builders are able to pack transactions into a block more efficiently, or it might be a result of the additional transactions they either add for value extraction or receive from private transactions. It is important to note that we are likely to observe an element of survivor bias as MEV blocks will only be proposed if they offer higher rewards than what the execution layer produces. As for the dip in the graph for non-MEV blocks with between 50 and 100 transactions, we are not sure the reason for this and welcome further investigation.

To conclude the transactions analysis, and since it wouldn’t be complete without some TPS figures, we also calculated average TPS pre- and post-merge with and without MEV.

Following what we have seen on the charts, MEV blocks have a slightly higher TPS compared to those without MEV.

Gas Used

We produced similar figures for the gas used per block and see a very similar trend. Notably, post-merge MEV blocks max out the gas limit more often than non-MEV, but the next highest peak is closer to the network’s target gas utilization than for non-MEV.

Block Reward

Block reward is the reward that a miner or validator will receive from transaction fees, in other words, it is the transaction fees minus the burnt fee; for pre-merge comparison, we have excluded the 2Eth block reward.

Pre-merge and post-merge we observe the distribution of MEV blocks is shifted right, which follows what we have seen on the transaction count histograms. In the case of post-merge the shift between MEV and non-MEV is greater which Aligns with the greater difference between the MEV and non-MEV transaction count distributions.

Transaction Fee

A block’s transaction fees are the sum of all transaction fees in the block, we calculated it by multiplying a block’s gas used by its effective gas price.

Transaction fees also follow very similar distributions as Block reward, which is no surprise given how they are related.

The number of blocks with transaction fees equal to 0 is greater pre-merge than it is post-merge; this is in line with the transaction count observations. Blocks with no transaction fees pre-merge have been excluded.

Rewards

Now for the bit you’ve all been waiting for: the rewards! What can a miner/validator actually expect to gain from selling their soul to the MEV gods? (I joke, I joke)

In the following chart, “Block rewards” continues to refer to the transaction fees minus the burnt fees, and “MEV rewards” is defined by the following flow diagram.

The rewards reflect many of the trends we’ve seen in the separate elements of the block structures. MEV returns fewer low-reward blocks and has a much longer tail; only 0.5% of the Block rewards are > 1Eth, whereas 3% of the MEV Rewards are > 1Eth.

To put the histogram into perspective, we have provided quantiles as well as min, max, mean and standard deviation of both Block and MEV Rewards. The maximum value of 278 ETH is erroneous and caused due to a known bug in the Eden fork that has since been fixed.

It did not make sense to compare the rewards pre-merge as these are much harder to accurately compute due to the methods that the MEV was distributed to miners.

Instead, we compared the MEV rewards over time. Here, the MEV rewards are in addition to the block rewards which act as a baseline for the total rewards. There is far more variation in the MEV rewards and they provide a consistent improvement to the rewards a proposer can expect to receive. Interestingly, we also observe an increase in the MEV rewards over time which is in line with an increased use of MEV across the network since the merge.

Conclusion

After so much anticipation, Ethereum’s transition to proof-of-stake went smoothly and has firmly cemented itself in the immutable history books. All over the world, a distributed army of validators is now working tirelessly to continue to secure the Ethereum network; it will be interesting to watch their roles evolve as the responsibilities of block building and block proposing grow apart.

During our analysis we’ve been able to compare the relative sizes of the old mining pools to the new staking pools, the impact of this on network decentralization, and the ability of staking pools to propose consecutive blocks.

We also found that MEV blocks are much easier to identify since the merge and the adoption of mev-boost is continuing to increase as it assumes the position of defacto MEV facilitator for validators. It was also very interesting to see how blocks built by MEV builders regularly contained more transactions and used more gas than regular blocks.

Looking to the future, there is still a lot to discover. There are insights to be gathered from categorizing the transactions in these MEV blocks to determine the reason for their increased number of transactions. Are they additional transactions extracting value? Are they from private channels? Something else? We would also like to find an explanation for why, when grouping and counting blocks by transaction count, we see a distribution with a different shape depending on whether the blocks contain MEV or not.

Feel free to contact us if you’ve any questions or feedback!

One more thing

Panda Metrics

We really enjoyed collecting and analyzing the Ethereum merge data, but there is still so much more data to visualize (Terabytes!) and such limited space in a blog post. We were saddened at the thought that the world may never get to see what we saw. Thus, we sought inspiration from the bamboo canopy and put our paws to work.

We decided to create a data visualization engine where people can see the Ethereum EL and CL rewards in an aggregated way. For the CL, you can see the average CL rewards per epoch for all validators of an entity and all epochs of a period. In addition, you can dive into the performance of each validator for a specific entity, as well as to see the detailed rewards received by a specific validator at every active epoch. On the EL side, we show the total rewards received by each entity, and normalize this by both the number of validators and the number of proposed blocks. Our data visualization engine is public and freely available for the whole community, and more importantly, we made sure it was dressed appropriately for the occasion.

Last but not least, it is important to mention that all the tools we developed for this work are available in our repositories (EL tools, CL tools, visualization).

Metrika Dashboards and Alerts

If you are interested in monitoring via dashboards and alerts the health and performance of individual validators and the broader Ethereum Beacon Chain network, you can visit the Metrika Platform.