Open Source Ethereum Infrastructure: A Beginner’s Guide to Essential Resources
--
This article covers essential resources for understanding the open-source software that runs the Ethereum network. Whether you’re a seasoned engineer looking to get involved with the Ethereum infrastructure or a staker eager to improve usability for yourself and others, these resources will provide you with the starting point you need to make a meaningful contribution to the Ethereum protocol implementation.
The teams behind the Ethereum protocol, including Arbitrum’s Prysm team, strive to provide end-to-end documentation in the form of published research papers on the theory, documented proof of concept and implemented functions, fully open-sourced implementations in different programming languages, and end-to-end tests that verify if the implementation is functioning as theorized. Most of this documentation can be found on Ethereum.org or the public Ethereum Github Repo, but newer changes to the specs may have their discussions and implementations across other links or forums such as discord or personal repositories.
Core Ethereum Specifications
- Ethereum Yellow Paper
- Vitalik’s Annotated Spec: Vitalik’s own summarized writeup and explanation of the reasoning for the proposed specifications on the implementation of the Ethereum network. It’s important to note here that Vitalik is merely summarizing the spec and reasoning in his own words, the Ethereum spec is written amongst the participants of the spec coming to an agreement.
- Ethereum Improvement Proposals (EIPS): Updates to the Ethereum spec are proposed publicly through EIPs; the ideas are discussed publicly through pull requests and the core contributors (including the proposer) discuss tradeoffs for implementation and inclusion.
Ethereum Node
An Ethereum proof-of-stake Node is the software anyone can independently download and run the Ethereum Blockchain. It consists of separate pieces of software that have different roles in running the blockchain — consensus and execution.
The consensus layer handles all things proof-of-stake (PoS), the mechanisms that allow each decentralized node to reach an agreement on the true state of the Ethereum blockchain.
- consensus client specifications — the official blueprint that anyone can read and contribute to for how the consensus software is implemented which includes all required functions for consensus services such as fork-choice logic, how blocks are proposed, how P2P needs to work, how bad actors are punished, etc.
- consensus spec tests — tests against all consensus spec function implementations to make sure it follows protocol.
The execution layer is in charge of executing Ethereum transactions between users and the applications that run on Ethereum (smart contracts).
- execution-specs — the official blueprint that anyone can read and contribute to for how the execution software is implemented which includes all required functions for execution services such as Ethereum virtual machine logic, transactions, and gas fees.
- execution-spec-tests — tests against all execution spec function implementation to make sure it follows protocol.
The Ethereum Foundation provides a roadmap with high-level concepts for achieving the community’s goals but ultimately depends on the consensus of core collaborators to drive the speed and direction of these implementations.
Research
Ethereum continues to evolve thanks to the work of many individuals around the world. A lot of high-value discussions occur in research forums and such as the ones below:
- Ethereum Research Announcements — summaries on the latest updates in research from the Ethereum Foundation, whether it’s an announcement of a new convention to allow in-person problem-solving or about a new release in Ethereum after core contributors reach consensus.
- Ethresear.ch — a semi-public forum for participating in Ethereum’s research efforts, but not for generic Ethereum discussion, specific EIPs, or ELI5s
- Ethereum Magicians — a discussion forum on the latest EIPs, thoughts & solutions on the ecosystem, and other issues to solve.
- Ethereum Protocol Fellowship Program Details — a program designed to onboard developers to the protocol development process, offering a stipend for selected individuals to do this work.
The Software Services that run the Network
The official Ethereum node clients are open-source public goods that implement the consensus and execution specs. These implementations are built by contributors from all over the world using different programming languages and with different backgrounds, many of which have no prior background in blockchain before contributing. Despite each implementation’s differences, they all adhere to the same protocol specs which add to the diversity and resiliency of the overall network.
1. Consensus Clients
In Ethereum, proof-of-stake consensus clients are responsible for how the blockchain reaches an agreement on the true state of the blockchain. The software handles importing validators, validator roles at each interval, block proposals, fork-choice, attestations, rewards, punishments, and the state of the beacon chain. It does not run the smart contracts or take in transactions, instead the execution client is responsible for all those features.
Below are the official implementations of the consensus specs (the blueprints for the consensus client software) in various languages, all with slightly different user experiences and feature support.
- prysm — Go implementation of the consensus specs maintained by the Arbitrum team with our docs found here.
- lighthouse — Rust implementation.
- teku — Java implementation.
- lodestar — Javascript implementation.
- nimbus-eth2 — Nim implementation.
Learn more about the role consensus clients play here.
Consensus APIs
REST APIs to interact with consensus client services. Each implementation of the consensus specs may have their own internal APIs that extend the feature set from the standard set.
- beacon-APIs — REST APIs supported by all consensus clients on their beacon node service.
- Eth Beacon Node API — swagger view of the beacon APIs.
- key manager-APIs — REST APIs supported by all consensus clients on their validator service.
- Eth Key Manager APIs — swagger view of key-manager APIs
2. Execution Clients
Execution clients are responsible for managing the Ethereum Virtual Machine, executing smart contracts, and providing interfaces for sending transactions. They also keep track of accounts. In order to function properly, execution clients rely on consensus clients to finalize the blockchain. Below are official implementations of the execution specs in different programming languages.
- go-ethereum — a part of the Ethereum Foundation and the original Ethereum client written in Go.
- nethermind — .NET implementation.
- besu — Java implementation.
- erigon — another Go implementation focused on optimizations.
Learn more about the role execution clients play here.
Execution APIs
JSON-RPC APIs allow Ethereum users to make transactions, call smart contracts, and allow other software such as the consensus client to interact with the execution client.
- engine json-rpc APIs — JSON-RPC APIs to communicate between the execution client and beacon node.
- execution-apis — JSON-RPC APIs to interact with the execution client and interact with the Ethereum blockchain such as posting transactions.
- Eth JSON-RPC Specification — swagger-like view for execution client JSON-pcs.
3. Supporting Libraries
Ethereum’s core development also depends on various libraries to provide specialized functionality such as cryptography, customized p2p, and customized encoding. (examples)
- BLST repository — provide the bls encryption used in Ethereum proof of stake.
- libp2p — Ethereum’s P2P library of choice.
- Prysm’s FastSSZ repository (fork of https://github.com/ferranbt/fastssz) — Encoding for P2P in proof of stake, this example is Prysm’s fork of the go implementation.
4. Client Diversity
The Ethereum network is more resilient with more diverse implementations so that a bug with one implementation does not take down the network. Dankrad, Ethereum researcher, writes more about this in his blog.
- Client Diversity — shows the percentage of usage on each client implementation by the overall network.
5. Supporting Tools
Besides the Ethereum Nodes running the blockchain, there are several other tools that support the end-to-end user journey of staking. The tools below are just naming a few.
- staking-deposit-cli — a command line interface tool that generates the validator key-stores, deposit object, and associated mnemonic to participate in staking. Vetted by external auditors and officially provided by the Ethereum Foundation; open-source as always.
- Ethereum | Launchpad — a website run by the Ethereum Foundation to provide an easier user piecer to send the 32eth and necessary data to become a staker. You may also interact with the deposit contract to do this yourself, however, this provides a safe secure way through a GUI.
- ConsenSys Web3 Signer — an external tool to manage your validator keys without importing into the consensus client that is supported by all consensus client implementations.
Supplemental Knowledge
This section overviews resources outside of core repositories, research forums, and core specifications where one can learn more about upcoming changes, newly proposed solutions, and the individuals helping to shape this amazing technology.
1. Meetings, Videos, Workshops
There are several places to join or watch public discussions in real time around the development, issues, and new innovative solutions in Ethereum.
Weekly Meetings
- Project Management — repo that publicly announces core dev calls, consensus dev calls, and implementers calls. There is typically a set of meeting minutes highlighting major discussion points. These calls happen on a weekly basis and are completely open to the public to listen in.
- Week in Ethereum — highlights major events for Ethereum during the week whether it was discussed on a call or on Twitter or elsewhere.
- Galaxy | Research Insights — provides summaries of various Ethereum calls.
Conferences
Many new ideas and bleeding-edge proposals are discussed at conferences through talks and workshops; the biggest being Devcon. Most of Devcon’s major talks and workshops are archived and publicly viewable on youtube. Another conference to keep watch of is Devconnect.
- DevCon | Archives — grouped archive page of all the talks highlighting the current work on the core Ethereum space.
2. Social
There are far too many Twitter accounts to list, but like most other news these days Twitter has usually been the place to hear new information around Ethereum development. This may also be the space to most easily reach out to other core contributors outside of conferences.
Discord
The discord messenger app has been the live chat app of choice for crypto development and Ethereum is no different. Most if not all Ethereum projects have a corresponding discord. Below just to name a few.
For other consensus and execution client discords please visit their respective web pages for official links, some may be through special invites only or part of their respective parent company discords.
3. Related Blogs from Core Contributors
Several core contributors have their own blogs that discuss design decisions, feature summaries, and implementation details in more depth. This is a short list and far from being comprehensive.
- Vitalik Buterin — Vitalik’s Blog
- Raul Jordan — Raul Jordan’s (One of the maintainers of the Prysm client and engineer at Arbitrum) Blog
- Dankrad Feist — Ethereum Researcher, writes his own thoughts and proposals on different aspects of the Ethereum Protocol.
- Adrian Sutton — Teku Lead, writes about some of the design decisions on their consensus client.
- Ben Edgington — PM on Teku, writes a book describing all the aspects of the consensus client in great detail, the book is WIP.
4. Contributing, Raising Concerns, Getting Help
As a member of the Ethereum community, you have the unique opportunity to shape the future of this powerful and transformative technology. By contributing to the open-source Ethereum infrastructure, you can make a real difference and have a lasting impact on the world.
As a developer, you can contribute code to help improve the Ethereum platform and create new and innovative applications. Or, as a user, you can report bugs and provide valuable feedback to help make Ethereum even better. No matter your skills or expertise, there is a role for you to play in building and improving Ethereum.
If you have questions or concerns about Ethereum, there are many resources available to help you get started. The resources listed above are a great place to start, with documentation and source code for the project. You can also join Discord channels or Twitter threads, or reach out directly to other contributors for personalized assistance.
It is important to note that to build a decentralized digital future based on values like transparency and fairness, we must understand the blockchain as a system that has been built deliberately and pragmatically, rather than as a black box.
Don’t miss out on this exciting opportunity to be part of the Ethereum movement. Join us in building the future of blockchain technology, and make your contribution today!
Happy Building and Happy New Year!
-James, Prysm Engineer at Arbitrum
