Exploring Ethereum’s Client Ecosystem
An analysis of Client Diversity based on a survey conducted in Q1 2023
We thank Tim Beiko for reviewing and providing feedback on this report and the ETHStakers community for encouraging validators to respond to the survey.
Key Findings
- The primary reason for node running is identified as ETH staking. However, a small number of users also run Ethereum nodes for data accessibility, and privacy, to become ecosystem contributors, and less than 1 % run Ethereum nodes for metrics collection.
- Over 95% of the respondents of the survey are running nodes for themselves.
- Geth is indeed the favorite client for the Execution chain followed by Besu, Nethermind & Erigon.
- Nethermind and Besu followed by Erigon turned out to be the clients of choice for switching if needed. Many users have shared not-so-good experiences with Besu since after the Merge. Interestingly, Reth is also being considered.
- Lighthouse is the #1 choice of respondents for the Consensus client, followed by Prysm, Teku, Nimbus & Lodestar.
- Teku & Nimbus are the preferred choices in case of a switch from the existing node-running clients. A few users expressed interest in Grandine.
- Geth-Lighthouse (21%) is the most popular client combination. Followed by Nethermind-Lighthouse, Besu-Lighthouse & Geth-Prysm at ~12% each.
- Availability of documentation along with the client/dev support is the main reason for the selection of clients to run EL/CL nodes.
- About 80% expressed concern with the present client diversity and ~25% of respondents listed “Client Diversity” as the main reason for the switch to another client for EL/CL node running.
- To increase client diversity, suggestions include standardization, campaigns, rewards & penalties.
Please note statistics shared in this report are based on the responses collected from the survey conducted by Ethereum Cat Herders in Q1–2023. Due to the limited survey size (<1% validators), the client diversity may NOT be an actual reflection of the present network share. However, users’ responses about their preferences, critical feedback, and improvement suggestions are valuable data points for the enhancement of existing Ethereum Execution & Consensus clients and eventually lead to an ideal distribution over the network.
Our sincere thanks to all users who responded to the survey. 🙏
Background
The Execution & Consensus Layer of Ethereum has multiple client implementations developed by different teams and written in different programming languages. Having multiple client implementations allow a healthy level of competition and encourages innovation among client developers.
The distribution of clients across the network isn’t ideal, especially with most nodes running the Geth client for the Execution Layer. Around 80% of survey respondents are concerned about the state of client diversity in the Execution Layer.
Client diversity reduces the risk of a single point of failure, making the network more resilient and less vulnerable to attacks.
We received responses from over 40 countries with the majority coming from the United States, France, Germany, and other European countries. We also received responses from users in South Korea, Japan, and Indonesia.
95% of respondents identified themselves as stakers, with the rest being non-stakers.
Ethereum node(s) running
Most of the respondents reported node running for personal use, while the remaining 7.5% of users running nodes for different organizations such as Aestus, Relay, Avado, Chainlink, Circle Data Science, Cryptocompare, Jump Crypto, Keynodes, Lido, Minipool Node, Rocket pool, Stafi, Stakewise, Swell, The Cryptocrats Partnership or personal projects.
As part of the survey, we asked users to select their primary reason or use for running Ethereum node(s) from four available multi-select options. The results showed that approximately 95% of users were running their nodes for staking purposes, while 46–53% of respondents were also focused on Decentralization/ Privacy and contributing to the Ethereum ecosystem.
Note: You may find the total of some charts over 100%. It’s because users responded with more than one option provided to them.
Client Diversity
According to the Client Diversity website charts, Geth is the majority client on Execution Layer and Prysm is the leader on the board with 58% and 39.8% network share respectively.
The stats. collected from our survey may not be 100% consistent with the Client Diversity website chart, but provides a pretty much similar idea of the uneven distribution of clients that we have on the Mainnet today.
Survey data suggests that a little over 50% of the respondents selected the Go-Ethereum (Geth); it indeed is the most popular client for the Execution chain followed by Besu (31.4%), Nethermind (30.8%) & Erigon (9.4%).
On the Consensus chain, Lighthouse (54%) is the #1 choice of respondents for node running followed by Prysm (23.3%), Teku (23.3%), Nimbus (15.7%) & Lodestar (1.9%).
As a reminder, the above chart (EL/CL clients on the network) is based on limited data collected from the survey with a sample size of less than 1% of node operators on the Mainnet.
Number of client nodes and EL-CL combinations on the Mainnet
In addition, our survey revealed that the majority of users are currently running a single client node, and only 3–5% of respondents are running more than 5 Execution and/or Consensus nodes.
Approximately 74% of respondents are currently running a single client combination, 23% running 2–5 client combinations, and only 4% are running more than 5 client combinations.
How important is it for you to run a minority client on the network? Why?
Over 75% of respondents mentioned running a minority client on the network is important for various reasons:
- It minimizes the severity of the slashing risk due to bugs in the majority of clients.
- It’s risk containment to avoid getting marooned if there’s a consensus-impacting bug.
- Support security and avoid penalties for a majority bug.
- The liveliness of the network in case of bugs is at stake.
- Don’t attest to a wrongly finalized chain.
- To promote decentralization/resilience, diversity, and redundancy. Keeping things honest, trust yet verify!
- Important for the health of the network, and to improve the ecosystem.
- diversity → protect my investment
- Makes me feel good that I’m contributing to diversity.
However, some users are hesitant to switch to a minority client due to the additional workload that comes with it. They prefer to stick with a working system, and a minority client may come with more issues and have fewer developers to support it. They would be more willing to run another client if there were well-written documentation available. Some users have considered running a minority client, but are hesitant due to potential problems and are waiting for better tooling to make the process easier.
While diversity is important, some users prioritize keeping their existing setup running smoothly as there is a lot of value at stake for them. They want to use the most mature, best-working, and most stable client available.
Highlighted users comments
- It is important to foster decentralization.
- It’s definitely important. Lighthouse was a minority client, when I started to use it. However, as large staking operators switched from Prysm to Lighthouse, I’m now running a majority client. I’m very much of the opinion, that large staking providers are the ones to hold accountable and urge them to switch as opposed to solo stakers. It’s still quite a hassle to setup a new staking rig and get familiar with a different client. Professional operators are better quipped to handle this.
- Running a minority client is important for chain stability: if the major client fails due to a bug, a minor client written in a different code base can keep the chain going. Ultimately, all clients should have similar shares. I would have preferred to run the failover node as a combination of minority clients, but some CL clients still have compatibility issues with failover support.
- Important, but currently all other execution clients are RAM resource intensive.
- If Erigon were the majority client, I’d still be running it since I need the smaller archive node size.
- Important, if my system would support them.
- Very important — the usual reasons. however, if the (somer) docs weren’t available, i’d have gone the popular route without a thought.
What impacts the selection of clients?
To better understand users’ thought processes, we asked users about the factors that influenced their decision to run a specific client. The majority of respondents cited Client team & dev support to be the most influencing factor followed by the availability of documentation, Specific feature, and majority bias. According to the responses, these factors play a vital role in the selection of clients, especially for new users.
Interestingly, many of the respondents reported having previously used Geth and Prysm clients but had since switched due to concerns about their super-majority status and the increasing diversity of available clients.
Switch clients?
As part of our survey, a few questions were added to collect the willingness to run a different client node and understand the challenges if any.
What would encourage you to run a different client than the one you are running currently — the missing element?
Based on the responses from our survey, it is clear that users value a client that uses fewer system resources and has good documentation, reliability, and fast sync. They are also interested in a minority client that has distinguishing features, such as lower storage/memory requirements, but want to understand any tradeoffs that may exist. Switching costs are high, so users would prefer a client that offers improved performance, stability, disk usage, or monitoring over their current client.
However, users also mentioned that the sync time for switching to a new execution client is a big hurdle, which can lead to procrastination. Therefore, a well-documented and fully functioning migration guide would be useful to make the process easier. Additionally, users would like to see incentives for running minority clients, even if they are small. Some users have tried running a minority client but felt like they were beta testers, which wasn’t worth the headache without incentives.
In case of switching to a different client, Nethermind and Besu turned out to be the Execution clients of choice, if needed. Interestingly, Reth is also being considered.
Nimbus & Teku are preferred choices in case of switching from the existing node running Consensus clients. Despite being a closed source at the time, a few users expressed interest in Grandine.
What features (that currently don’t exist) would you like to see an EL/CL client implement?
According to our survey respondents, they would like to see better incentives for individual node operators, such as proof of solo authenticity, to encourage more participation in the network. Some users suggested that they feel the separation between EL and CL is unnecessary and that the clients should handle both layers. They also suggested that combo EL/CL software packages would provide a vastly better user experience and could help more inexperienced users become home stakers.
Other features that users would like to see include pruning (although it already exists), simpler or inbuilt monitoring, Beaconchain-like features for earnings, tax forms for staking income, automatic client switching, data management and clean-up, improved and informative logging on errors and sync status, MEV smarts and other profit maximization, alerting capabilities, and reduced hardware requirements.
Some users suggested that they would like to see a unified EL/CL application that handles both functions in one application, as the current format can be clunky. Faster sync times on the EL side would also be welcomed by users, as switching clients without redundancy or failover configurations can result in a downtime of 1–2 days.
Lastly, some users would like to see a proper user interface for validator management and reporting, as they feel it is not enough to rely solely on Beaconcha.in. Additionally, users would like better access to and display of internal stats.
EL-CL Client packages & testing
- Feel the EL/CL separation is unnecessary. The clients should handle both layers.
- A ‘plug and play’ software package for EL/CL combinations. One application that handles both functions. [Multiple mentions]
- Automatic client switching.
- A temporary failover option for CL when a given EL is offline.
- How about the client host a web server at some port that I can just go to and find out information about its state — CPU, memory, and other statistics? I have to do the whole Prometheus, graphana garbage. and the stats don’t even mean anything to me.
- One-click to run the whole darn stack.
- Differential fuzzing between client pairs.
- Proof of Keys test (dry run to withdraw for example). On-chain encrypted emergency backup of keys. Graffiti plugins
- Single binary
UI, monitoring, and other features
- Simpler or inbuilt monitoring.
- More graphs and interfaces, an easy dashboard
- Graphical interfaces.
- Improved and informative logging on errors and sync status
- Alerting capabilities
- Auto update on startup
- vastly better UX
- CL Team provides a nice Grafana dashboard
- 1-word command updates, more ease of use.
- Not interested in features. They should be as resource efficient as possible, maintaining themself configurable in a closed loop,
- Monitor SSD space: currently LH only sends data about local disk, and not data-dir.
- A proper UI for validator management and reporting. Not enough to rely on Beaconcha.in all the time.
- Better internal stats access and display
- it would be nice to be able to run on non-super-enterprise-grade hardware & NVMEs
- Native Web UIs
- Better logging, stats, and other on-chain analysis
- Browser-based UI and toolset
- Would be good for clients to report their vendors to validators, this way everybody would know the exact number and proportion of different clients.
- Better troubleshooting
MEV boost
- Built-in MEV boost, MEV smarts and other profit maximization.
- The clients should consider local mempool before accepting bids from relays. It might rarely trigger any change, but just as a matter of principle, the relays (especially if they’re censoring) should only be able to buy the blockspace if they outbid the mempool.
- There should also be a feature that says relays have to outbid mempool by at least x ETH before the client will accept it.
Database
- online state pruning; better database compression. pruning of ancient blocks
- store chain data in a SQL database for easy querying. No, graphql is not good enough
- Some kind of internal checkpointing, e.g. on a weekly basis, would let you roll back from database corruption without a complete resync!
- Data management and clean up
- I would like the duties to be sent to different CLs depending on whether they are Att, Blocks, or Sync committees. Also, we could go back to the N:N model between CL/EC. that would help a lot in redundant systems.
Execution clients specific
- Faster sync times on the EL side would be great (i.e. dropping the load of storing/validating historic blocks/states). If I had to switch clients without redundancy or failover configurations, there’s going to be a downtime of ~1–2 days which is not the end of the world, but still suboptimal and thus a barrier to change (at least for the EL).
- Faster sync times (something like checkpoint sync we have in the CL). Better reliability against sudden crashes (reduce risk of DB corruption). Reduced DB size.
- Multi-EL clients for failover
- Self-prune in other EL clients, similar to Bonsai Trees in Besu would be great.
- An easy health check
- Execution client on HDD
- Getting added to Linux kernel. DB corruption protection.
- Reduced hardware requirements. Lower memory / CPU / disk usage.
Reason for switching to another client
While self-pruning was greatly appreciated, Besu users reported significant problems after the merge and were unable to get it to run reliably, losing more than 50% of attestations over several days of use. They also noted poor performance and stability issues. Erigon users reported difficulties with continuous updates. As for execution clients, most users switched to Nethermind due to its better stability and lower resource consumption.
Other highlighted users’ comments for the reason to switch to another client are:
- I choose based off of ease of use and TB footprint for an archive node.
- Switched from Geth to Besu for client diversity. Switched back to Geth after the merge for stability.
- Switched from Geth to Erigon/Besu to reduce supermajority risk.
- Earlier Geth, switched to Erigon because it’s much faster, uses less space and Geth has a super majority.
- Originally ran Geth and Prysm. Switched to Lighthouse for diversity and everything worked fine. Tried switch to besu for diversity but was unable to get it to start syncing and felt it lacked documentation. Then tried Nethermind and while I was able to get it to sync, I had frequent missed attestations (about 4 per validator per day) and low 90% effectiveness according to beaconcha.in. I upgraded my equipment (from 8GB of RAM to 32GB RAM and from a 2 TB sata SSD to a 4 TB nvme SSD). I still had the same issues. I switched back to Geth and had no issues with very, very rare attestation misses and 99% effectiveness.
- Main machine runs Geth/Prysm. I tried with Besu/Prysm. Switched back to Geth due to missed attestations daily with Besu. Besu continues running on my back-up machine. Another back up machine runs Geth/Lighhouse. Reason for 1 Main + 2 Backups is to have maximum client diversity.
- Switched from Geth to Besu to improve client diversity. Switched from Besu to Nethermind for better stability, lower resource consumption (at the expense of manual prunning).
- Used Besu but post-merge it missed several attestations a day and produced blocks with zero transactions included, so switched back to Geth.
- I used besu but after the merge it suffered significant issues and I could not get it to run reliably again (losing >50% attestations over several days of use)
- Initially Lighthouse and Geth. Switched to Nimbus and Besu because of client diversity, but later ditched Besu for Nethermind after numerous issues during the Merge.
- Earlier used Geth and Prysm, switched Besu-Lighthouse when autopruning was added. Reason for switching was to enhance client diversity
- Earlier used Prysm. I switched because the load on the system was too high.
- Switched from Geth/Lighthouse to Besu/Teku before the merge to avoid using majority execution client after merge. Also Lighthouse was in some metricas over 33% at that time.
Stand-by system is geth/nimbus on a less powerful machine also used for gathering some statistics of the network/daily balances.
Client interaction frequency
We asked users how often they interact with their EL/CL clients. The results show that approximately 42% of respondents interact with their clients every week, while 40% interact during network upgrades. Around 13% of users interact with their clients daily, and a small percentage, ~3.8%, interact with their clients multiple times per day.
What do users think of their clients?
With unique features and h/w — s/w requirements, Ethereum node runners have multiple options available to select the client. From the survey, below is the list of merits and improvements requested by node runners. Here are some key highlights of what users think of each client.
Execution clients
Besu
Validators and node operators currently using Besu, appreciated the client for
- auto pruning,
- stability,
- ease of installation,
- documentation,
- a trusted & active dev community,
- a responsive team,
- regular updates,
- professional support, and
- having a minority client implementation with no common code with Geth.
However, users [20+ mentions] reported stability issues, slow syncing, missed/wrong attestation, low performance, and the need for better documentation.
Reliability was poor following the merge, but much better now.
Technical issues surrounding the merge were not well communicated.
Documentation could be improved and performance tuning tips can be explained more clearly and frequently.
Makes me nervous during hard forks (besu)
Besu has come down like 4–5 times post merge that’s required a re-sync
Took forever to stabilize and stop missing attestations.
Slow syncing, missing attestations several times per day
Bugs. Java.
Besu’s memory usage and leaks.
Frequent RocksDB corruption issues with Besu
The performance of besu is lower than of Geth, especially for large blocks. Leading to a few wrong attestations because blocks were not validated in time.
I still miss one attestation every few days, and missed about 10% sync committee participation
Besu — some releases are unstable. Long time to fix issues.
I wish there was an in-line command interface similar to what Geth has.
Besu has a lot of problems, currently with open issues and a month of downtime.
It was fairly difficult (Compared to Geth) to get it configured and running. There was lots of fiddling around, Googling error messages and asking devs about things on Discord before it finally worked.
Erigon
Erigon is highly appreciated for
- Less TB for archive, light archive node,
- archive node features for data analysis,
- fast sync, good custom APIs for querying blockchain data small archive node.
- Easy to monitor fast restart.
However, a few users [5+ mentions] reported that Erigon
a lot of RAM usage
requires large disk space,
too many changes and resyncs every few months.
Sometimes the database gets corrupted and I have to resync fully which takes days.
Erigon is stable up to version 2.34.0, at 2.36. it is stable again but takes a lot of disk space.
Geth
Geth is the #1 choice of node operators. The main reasons as highlighted are
- Easy to use, good documentation, very effective, de facto standard.
- Excellent, trustworthy dev team with a long proven record;
- Preference for Rust/Go
- Better documentation & guides in comparison to other EL clients
- Geth is stable and takes a low-space disk.
- Reliability [10+ mentions] and low hardware requirements
- Really good Prometheus metrics
- Rare bugs & Issues
- Stable Geth-Prysm & Geth-Lighthouse Client Combination.
- Stable, quiet, peace-of-mind operations; hardened, secure software; low frequency of updates, yet critical patches will be fixed immediately.
- Performance, and support of some Geth-specific RPC
But, at the same time, users are also worried about the issue of being a majority client and slashing/penalties risk associated with the bugs in the client. Other reported concerns are:
No online or auto-pruning [5+ mentions]
Geth sometimes gets random and inexplicable database corruptions that require a full resync
Geth corrupts easily with shutdowns, and restarts take a long time to synchronize.
High storage requirement.
Geth: disk footage. 2TB Nvme ssd is quite expensive and 50% of it is used from day 1 for the node.
Having to migrate to a 4TB ssd within the next 2 years would be unfortunate.
No GUIGeth: Too many nodes running, but it’s still the most viable option.
Very IOPS dependent trying to prune geth the first time messed up the whole db. had to resync from start
Geth is very sensitive to unsafe shutdown [5 mentions] and takes a long time to heal and takes hours slow to add features.
Nethermind
Multiple users shared that Nethermind is great for simplicity and ease of use as compared to other EL clients.
- Nethermind is easy to install,
- has regular updates
- is compatible with most of the Geth scripts and methods.
- auto-pruning
- easier to use fast sync,
- good custom APIs for querying blockchain data
- Nethermind is quick and has great support.
- Developers' timely response to bugs & problems.
Nethermind is the most popular client after Geth on Execution Layer. However, there is some room for improvement. Multiple mentions to improve memory usage and pruning issues. Related feedbacks are:
Not sure how to prune it [7 mentions], it begins to be quite big.
Nethermind uses too much memory [5 mentions], don’t like the .net ecosystem either, but it works.
RAM consumption and heavy CPU usage on initial sync, but I put 64GB so its not a problem. [3 Mentions]
The lack of documentation in the docsNethermind has somewhat confusing documentation. [3 mentions]
Logs are hard to read, it’s not colored.
The community is not as big as the one from the other clients, it’s sometimes hard to get support
Prune time
Online pruning does not seem to work well for Nethermind. Considering switching to Besu (and Bonsai), but have seen many complaints about bugs in Besu, so will wait for now.
Consensus Clients
Lighthouse
Stability, reliability, and dev team [20+ mentions] are highly appreciated by users.
- Lighthouse is rock solid. They make me feel fuzzy when they fuzz their code Stability, ease of installation, and documentation are rock solid, they just work.
- Simple and generally stable and performant.
- Lighthouse is developed in Rust
- not a majority client, popularity is growing, need to switch again
- Stable, quiet, peace-of-mind operations; hardened, secure software; low frequency of updates, yet critical patches will be fixed immediately.
- The Lighthouse team is awesome, every time I have a question, there is someone to answer me
- Rust. I trust the sigma prime team, I have high confidence in the product, and it’s the client I am most able to operate.
- Easy to maintain, update notifications/explanations on Discord (they only tag @everyone when there’s a new update
- Has a good reputation in the community, not the majority client (if Lighthouse became the majority client, I would switch)
Though some users have
future majority concerns
The name lighthouse is not unique so searches get polluted with irrelevant results.
Not being able to download the latest, but having to put a specific file
Using DockerConfigurations is sometimes buried in discord,
no easy way to upgrade versions.
No GUIvery IOPS dependent
I don’t need a GUI/TUI, but I think it’s a valuable feature that ought to be provided by all consensus clients.
All of the clients are still out of reach for novice users.
Graphical solutions like dappnode ought to be developed upstream and made accessible to every client.
One sudden crash of the database led to missing proposal.
Nimbus
Known for light node, light on resources.
- High reliability
- Great documentation and rock solid
- Nimbus is lovely, and
- also has low resource usage, disk usage
Some users are
concerned about state issues
Logs unclear. I miss a few attestations every day, but I don’t understand why
releases and release notes seem random,
sometimes breaking features get added randomly with no mention in the release notes.
Prysm
Reported to be
- Stable,
- plenty of documentation,
- ease of upgrade
- been rock solid, they just work
- key manager API
- auto-update when launching
- Prysm had a great product early in the Eth2 development cycle and had better support then. Now I appreciate the metrics they have.
Apart from being one of the leading clients, Prysm received very few concerns as
it may not be as efficient with disk space or other resources.
That I had to learn Linux all over again after leaving it for about 15 years
Teku
Many users switched to Teku for client diversity. It being a minority client was the key motivator along with others like
- Teku and Besu work very well together,
- Regular updates
- professional support
Reportedly, Teku
is not implementing state/validator_balance for arbitrary epochs for computing daily profits (lighthouse and nimbus do).
A lot of RAM usage
no useful entries in logs / seems that Teku is overwhelmed in busy network situations.
Intermittent high CPU usage
What can be done to increase client diversity?
Most of the users are on board with the importance of multiple client implementations. We received multiple suggestions to increase client diversity. Good support and client documentation are important. The community groups like EthStaker as other avenues to get information are also helpful.
Easy switching
Make switch ezpz for noobz
Easy switching, and easier tooling for setting up and migrating existing EL/CL pairs that bake in diversity could be helpful. According to a user,
Honestly, switching clients is already too easy for this to be taking so long.
However, some suggest that having specific switching guides, for combinations of different setups eg Dappnode, Rocketpool, etc. will be nice to have. Node operators want to be assured that it’s not going to break if left alone for a few weeks.
- More 1-click install staking solutions, with enough credibility to encourage institutional adoption.
- Improve docs, provide stats for attestation and proposal performance, and show stats for CPU, disk, memory, and bandwidth usage.
- Execution clients catch up to Geth in effectiveness.
- Improve the robustness of minor ECs and add some graphical interfaces.
Standardization
Each client is running their own chain database (RocksDB, LevelDB etc.), and there is no interoperability between them. I.e. Besu cannot use the same DB as Nethermind. With a standard database schema shared across multiple clients, one could have the clients switch automatically, continuing recording the blocks, based on the current diversity spread. Although this is unlikely in practice.
- Make swapping clients easier through more standardization. Likely not possible, but if installing, configuring, and running clients were more similar, people might be willing to trade out one client for another.
- Standardize confit/command line values
- A standard database format or a DB conversion tool?
- Some features/tools that reduce or eliminate resyncing would encourage change.
Rewards & Penalities
According to a few respondents incentive matters. Having economic incentives for running minority clients may provide better encouragement. The suggestion is to change the way validators are selected for execution/consensus rewards, so users make more if they run a minority client and that’s compensation for taking a risk. Minority clients should get higher staking rewards. A few others suggested penalties for running staking nodes with a client with > 50 % market share (e.g. Geth).
A few users think that having more POAPs/NFTs rewards, and giving badges may help promote client diversity.
Campaigns
Some sort of campaign, and good people like Superphiz :)
Conclusion
Overall, these survey results provide valuable insights into the motivations and demographics of Ethereum node runners. Having a better distribution of different clients with no single client dominating over 50% of the network is critical for the security, stability, and resilience of decentralized Ethereum networks.
Furthermore, it helps avoid scenarios that can result in disruption, monetary loss, and centralization, which can be detrimental to the network and its users. Therefore, it is essential to strive for a well-distributed client diversity across the network to ensure its smooth operation and safety.
PS: The report is updated with minor edits. Appreciate feedback from Besu team to suggest edits.
