Prysmatic Labs Ethereum Monthly Development Update
6 Months Since ETH Proof of Stake Launched
Team Growth
This quarter, we have been focused on growing our team. We have recently onboarded more developers into Prysmatic Labs, with Kasey Kirkham as our first hire of the year. We are thrilled to have Kasey joining us as he brings a wealth of experience in software engineering. Kasey has spent his career building distributed systems at startups with interesting missions, including Niantic, Planet Labs, Voxy and Topsy (Prysmatic is his 5th employer with “Labs” in the name). He joined Prysmatic Labs because he wants to help the Ethereum network reduce energy requirements through the transition to Proof of Stake.
Learning from the First ETH2 Mainnet Incident
On April 28, 2021, we had our first, real incident on Eth2 mainnet, which prompted a long weekend of debugging and resolving the problem as professionally as possible from our team. At epoch 32302, the beacon chain started missing a lot of block proposals. Prysm was the likely suspect since Prysm represents a large portion of Eth2 operators. After a short while, we were able to reproduce the error locally and the problem was indicative of a known issue regarding Eth1 data voting and validator deposits. Across the span of 2 days, we were able to arrive at the root cause and resolve the problem carefully for all stakers running Prysm. With real money at stake comes a lot more responsibility than we had before, and we took this as a huge learning opportunity to improve our internal processes and communications to all users of our software.
Things we have improved upon since the incident:
- Deeper inspection into cache-related features in our codebase
- Better process flow around release pipelines and communications to users in times of panic
- Greater emphasis on defensive programming practices, being more stringent about code review when it comes to sensitive changes to our code
Things we are still working on since the incident:
- Improving our observability pipelines to fully grasp information we need to catch issues early and have full information when investigating issues
- Adding more guardrails in the form of static analyzers to our codebase to prevent similar bugs from ever entering our project
R&D Updates
The Eth1 <> Eth2 Merge
The most important talking point regarding Ethereum’s future this year is the concept of the Eth1 and Eth2 merge. In order to upgrade Ethereum to proof of stake, there are multiple things we need to keep in mind:
- We already have a proof of stake chain live with real ETH at stake known as “the beacon chain” that cannot go down
- We have an existing proof of work chain with billions of dollars worth of critical applications running that cannot go down
- We have some really experienced teams that are experts on proof of stake consensus and teams that are experts on EVM execution
- We can “merge” the two by having Ethereum use the beacon chain proof of stake for consensus, and EVM execution for transactions
“The Merge” is referred to as the collective effort of Ethereum switching to using the beacon chain for consensus while using traditional eth1 runtime with the EVM, transaction pools, gas, and other parts we are all quite familiar with. Although not pretty, this will be the most important milestone to date regarding the base layer of the Ethereum blockchain. Needless to say, it is a complex endeavor, and all teams are heads down on making sure this is done securely and seamlessly.
This past month, Protolambda from the Ethereum Foundation pioneered a hackathon called Rayonism for all interested parties to rapidly prototype code towards “The Merge” in a quick sprint fashion. Rayonism culminated in a test network called Nocturne, which featured many different combinations of ETH proof of stake <> ETH execution clients running a real beacon chain with transactions as desired. This was a great way for all teams to gauge the complexity of such a task. Prysm participated in the nocturne testnet by running a good portion of the network, and the chain advanced as expected.
This summer, we are focusing on the first hard fork of Eth2, and in the meantime, working hard towards solidifying the work we’ve done so far on the merge so that we can all work more on it within the coming months.
Sharding
After the merge, there comes the shards. Sharding provides data availability for layer 2 applications to strive on. There has been tremendous progress for sharding on two fronts: spec improvements and prototypes. On the spec front, beacon chain specification refinements continue, data availability designs for the beacon state are getting optimized, and the processing of sharding has improved on efficiency. On the client teams’ end, both Teku and Prysm have a prototype branch that follows closely on the sharding spec and actively provides feedback to researchers. On the networking side, we have been working on subnets for shard blobs and the discovery protocol. A separate software process that can track shard data and validator confirmations is getting worked on by Protolambda and we also plan to help out on that front. This would be useful for rollup testing. If you are interested in this bleeding edge, please let us know!
Incoming Features for Prysm Users
First Beacon Chain Hard Fork: Altair
Coming up next is the first hard fork for the Ethereum proof of stake beacon chain: Altair. Altair brings a lot of long-awaited improvements that were not available at the genesis of Eth2. Many of these are optimizations, but it is also the first time light-client support is added to Eth2. After this hard fork, it is possible to build very lightweight nodes that could run in browser wallets or phones to validate only the required parts of the beacon chain and provide many awesome features not possible today.
Teams plan to launch a testnet for the hardfork within the coming 2 months as a first step towards ensuring safety of the upgrade. Network upgrades in distributed systems are indeed risky as they could lead to serious problems such as contentious chain splits if a certain population of node operators disagree with the changes in the upgrade. Thankfully, the Altair hard fork is non-contentious and should be 100% regarded as a network upgrade that will address efficiency concerns of the current beacon chain. You can find more details about what’s included in Altair here and an annotated specification created by Vitalik Buterin himself is maintained here.
Official Support for the ETH2 Standard API
Prysm has had gRPC API endpoints for almost 2 years which helped a lot of early tooling get familiar with Eth2 and build for it, such as block explorers. However, this API does not follow the standard set by the Ethereum Foundation, which is a standard JSON POST/GET set of definitions. To support this, Prysm needs to use a special middleware known as a gprc-gateway to convert incoming and outgoing JSON requests into the gRPC standard we use today. Unfortunately, this was no easy task, and took us several months before we could get it in a manageable state.
While we have had support for JSON POST/GET endpoints through the grpc-gateway plugin extension, it did not support the features we required to align with the spec. After a lot of deep digging into grpc, and the grpc-gateway plugin, Ivan and Radek from our team have successfully replaced our APIs which had previously relied partially on some smaller, third party libraries, to using the entirely native and official protobuf v2 APIs from Google. This finally unblocks us on our Eth2 API standard implementation, and we are hoping to have it ready for production usage by the next release.
Please note: Our implementation of the standard Eth2 API does not currently support the eth/v1/validator/ endpoints. Because of this, Prysm validators are not yet compatible with beacon nodes provided by Infura. Please keep this in mind for any fallback nodes you attempt to set up.
State Checkpoint Sync (Weak Subjectivity Sync)
An important feature of Proof of Stake in Ethereum is the concept of “weak subjectivity”. Although it’s a very heavy word, it refers to mathematical guarantees that allow us to synchronize the chain from a “checkpoint” that isn’t the genesis state. That is, nodes can sync from a finalized checkpoint far enough in the past that we have enough security guarantees of chain consensus for beacon nodes. Not only does this make synchronization of the chain extremely fast, but it also helps us prevent certain attacks known as long-range attacks in proof of stake. If this sounds like magic, there are a lot of explainers about why weak subjectivity works and details about its security here. Our team is currently working on adding weak subjectivity synchronization into Prysm securely and you can track our progress here.
Dedicated Support for the Prysm Web Interface
An important feature for many Prysm users is support for a web interface to manage some aspects of their validator. This web interface is especially useful for stakers that are less familiar with how to deal with command line interfaces and would prefer a point and click experience. From speaking with many users of the interface, many would like to see it become a lot more useful and become a complete alternative to the command line. Due to limited team bandwidth, we have not been able to prioritize the Prysm web UI as much as we hoped. However, we recently set out to find a frontend engineer that will help us push the envelope on user experience for validating on Ethereum and hope to onboard a new teammate soon!
Interested in Contributing?
We are always looking for devs interested in helping us out. If you know Go or Angular and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you.
Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.
As always, follow us on Twitter or join our Discord server and let us know what you want to help with.
Official, Prysmatic Labs Ether Donation Address
0x9B984D5a03980D8dc0a24506c968465424c81DbE
Official, Prysmatic Labs ENS Name
prysmatic.eth
