Prysmatic Labs June Ethereum Development Update

New Team Member

We recently added a new teammate, Zahoor Mohamed! Zahoor recently shipped Ethereum Swarm to mainnet, marking a significant milestone in decentralized file storage. Swarm is incredibly decentralized at the time of mainnet launch, and Zahoor brings to our team many years of distributed systems knowledge. Since joining, Zahoor has been working on key optimizations to Prysm’s underlying data structures and brings a great perspective towards improving our client.

Upcoming Altair Hard Fork

What you need to know

The first network upgrade for the Ethereum beacon chain, Altair, is happening in the coming two months. When the Ethereum beacon chain was launched, it came with significant risk to early adopters as it is a very new technology. To ease concerns, some of the slashing parameters were relaxed at launch. The Altair hard fork is meant to tighten up some parameters of the chain as they were meant to be for security reasons, introduces some important optimizations for accounting validator rewards and penalties, and adds a much awaited feature of light client support. Light clients on Ethereum proof of stake are much easier to run and will be far more practical than they are today.

Changes coming in Prysm v2.0.0

Altair is a hard fork, meaning your node software needs to be up to date ahead of the fork to remain on the chain and not lose any validator rewards. The Prysm version supporting Altair will be v2.0.0: a major version upgrade. We’ll explain what improvements are coming in Prysm v2 which we believe will seriously improve the quality of our project and make it more focused on developer experience. Prysm v2 will not break most existing setups of how to import keystores, setup Prysm beacon nodes and validators. However, we will be doing the following changes which mostly affect developers and contributors to Prysm:

• Restructuring the package structure of our Github repository for more standard Go conventions
• Removing unrelated packages and code that is not directly related to running a beacon node or validator client from our repository
• Removing all deprecated flags from the project
• Creating a more maintainable code structure that leverages semantic versioning for future hard forks

We will be announcing Prysm v2’s estimated release ahead of time to all node operators via all our communication channels, including Medium, Discord, and our mailing list

Why Altair Was Difficult to Implement

The Altair hard fork was notoriously difficult to implement in Prysm. Go, as a programming language, is notorious for the lack of generics at the time of writing. Writing maintainable code is about reducing duplication and ensuring it is easy to understand and easy to test. Many of the items in Altair involve key, drastic changes to data structures such as blocks and the beacon chain state, which would break our existing code. When writing Altair, we had to come up with a futureproof strategy that will make supporting new hard forks easier than the first time. Some of the highlights of our progress are:

• Add new Altair objects in protobuf format. These are beacon state, beacon block and the new sync committee objects. Note that replacements are not backward compatible with the existing object
• Update beacon block’s attestation processing to account for the new incentive reform. Validator’s attesting status is now updated in the beacon state by participation flags on per slot basis
• Add beacon block’s sync aggregate processing to account for the new sync aggregate object in block. This ensures that the proposer and sync committee participants are getting properly rewarded
• Update epoch processing for the new inactivity leak changes and epoch participation using the participation flags from per slot. Sync committees and participation flags also get rotated
• Add the upgrade function to translate existing beacon state into Altair version beacon state. This is arguably the most important piece of code, where the function inputs the phase 0 version state and returns the Altair version state
• Add the new network pipelines for sync committee objects. These pipelines are essential on defending against dos attacks and regulating the p2p network into the healthy state
• Add the new RPC endpoints. The post-fork validators will be using these endpoints to complete the existing duties and the new duties
• Add proper optimization such as caching around the new sync committee. This is needed to alleviate extra beacons state lookup and validator shuffling
• Pass the Altair spec tests. Special shout out to all the researchers that worked tirelessly on it. The spec tests have been an integral part of the client’s success

Prysm Slashing Event and Our Response

Summary of Events

Exactly 7 days ago, a validator using Prysm was slashed. The user posted a detailed report on reddit.com/r/ethstaker explaining how they were running Prysm and the sequence of events. The occurrence resulted in the user losing approximately 0.2 ETH and getting kicked out of the validator registry, missing out on rewards until they are able to withdraw their remaining balance. After some analysis, we identified the root cause.

Root Cause

As usual, the most likely cause for a slashing event was running the same validator key in two different processes. In Prysm, we currently have a lock in the validator database. This means one cannot run the two validator processes with the same database on one computer, and we mistakenly believed it would be improbable for a staker to accidentally do this. Recently, we released a new command for Prysm called `client-stats`, which lets validators run an optional, sidecar process to collect metrics from Prysm and report them to block explorers. One of our stakers was running Prysm in Linux as a system service with a set of flags that looked something like this

prysm.sh validator --datadir=/somevalidatordatafolder --verbosity=debug …

The staker, however, wanted to try out our new client-stats tool and while the Prysm validator service was running in the background, accidentally ran the following command:

prysm.sh validator client-stats --someflags

The user meant to type `prysm.sh client-stats` but the keyword validator slipped in accidentally. Our command-line library, urfave-cli, a popular choice in Go, would ignore the word `client-stats` because it is not a subcommand of the keyword validator. This means the user essentially spun up a new validator instance using the default database path, meaning he now had two validator clients running different databases on the same machine but with the same private key. The user was soon slashed.

Our Response and Resolution

First of all, we want to apologize to the unlucky Prysm staker that met this unfortunate fate. Although we can enforce different kinds of safety via command line flags and tooling to avoid someone running the same validator key on the same machine, it is not the best solution to the problem. The best way to realistically prevent slashing events is to enforce what is known as doppelganger detection. This means the validator client will actually use on-chain information to detect if the same validator key is actively attesting/proposing elsewhere and forcefully halt the process. This means a validator can protect itself from getting slashed even if the validating key is running on some other computer. This is becoming a standard feature among client teams, with the Nimbus team pioneering the effort. Since this slashing event, we have taken the following actions:

1. Implement doppelganger detection for Prysm validators, available in the upcoming release
2. Adding additional system-file locking around the validator wallet, preventing even more accidental, although improbable, scenarios
3. Return error on unrecognized arguments when running default commands in Prysm, which is not handled by default by our CLI library used in Go, fixed here

We believe doppelganger detection is our best bet at removing the risk of accidental slashing as much as possible for Prysm operators.

What’s Next for Q3

Planning for the Merge

As Altair looms on the horizon and our team wraps up most of the key feature work for its release, we begin to look forward into the proof-of-work to proof-of-stake transition, known as “the merge”. Earlier this year, back in May, the different core development teams worked together on prototyping a quick merge testnet, which would showcase an Ethereum beacon chain running alongside Ethereum execution clients, such as go-ethereum, to execute transactions. Essentially, this was Ethereum as we know it running on proof-of-stake. Although this proof of concept was really cool and showed us what’s possible, it left many questions unanswered. In particular, Q3 will be important for the different teams to come together on a strategy for how exactly the transition process from PoW to PoS will be done seamlessly, the user experience, and remaining technical challenges. Fortunately, there are no remaining research challenges on this front, so the remaining items are all implementation and user experience related. Prysmatic will shift to an almost all-hands-on-deck approach after Altair to get us finishing the work on “the merge” with good timing.

Sharding Work

A lot of research work on sharding has been happening recently. The sharding specs are constantly improved. During the biweekly, eth2 implementers call, Protolambda from the Ethereum Foundation explained there are advantages for the sharding design to implement similar market space between shard block builder and shard block proposer. The proposal was then reflected in the draft PR and has gotten much feedback since.

On the implementation front, Teku is on the cutting edge with the sharding prototype beacon node. Us at Prysmatic Labs will be working on catching up to sharding prototype beacon nodes in Q3. Many details of sharding p2p networking are still under discussion, such as gossipsub propagation of shard headers, gossipsub subnets for shard blobs, and the discovery of subnets which still need to be defined. We are particularly excited about the independent design of a shard node which specializes in tracking consensus via a beacon node, can sync shard data, and can serve an API for layer 2 solutions. This is useful for debugging and rollup testing, the sooner we can direct the sequencer to this API, the sooner we can build the first Ethereum proof-of-stake ready rollup.

Reducing the Use of the Term “Eth2” for End Users

TL;DR: We believe the term “eth2” is fine to use between core implementers and researchers to refer to the set of upgrades that will bring sharding and proof of stake to Ethereum, but non-technical investors, users, and dapp developers should only be aware of the term “Ethereum’’ to describe the Ethereum blockchain. We want to minimize confusion, fragmentation, and prevent scammers from taking advantage of a non-existent difference between ETH and ETH2 cryptocurrencies after the merge.

We recently posted about our desire to minimize the use of the term “Ethereum 2.0” in our project and we want to shed more light on this decision. Over the past few years, the set of changes being done to Ethereum’s consensus and its future adopted the name “Eth2” or “Ethereum 2.0” as a convenient way to separate it from the current work done on Ethereum core dev, which mostly concerns proof-of-work clients. As time went on, the name has become quite popular. However, as we get closer to “the merge” of Ethereum proof-of-work and proof-of-stake, we want to question the utility of the term. We are very concerned for newcomers to Ethereum questioning why there is an eth2 after the merge is completed. We believe after the merge:

• There is no Ethereum 2.0 beacon chain, there is only the Ethereum beacon chain
• There is no Ethereum 2.0 cryptocurrency, there is only ETH
• There is no difference Ethereum 1.0 and 2.0 for end users
• Ethereum 2.0 core dev is just Ethereum coredev

In essence, we believe it is in the best interest of the future of Ethereum to just call it what it is. We want to avoid anyone asking “should I buy this ETH2 coin on this exchange vs. ETH?”. The way people interact with Ethereum post-merge will be identical to Ethereum today. Using the term eth2 for end users implies there is a difference between eth and eth2 when there is none after the merge. Making it seem like there is a new asset enables scammers to create a dangerous user experience for beginners.

We understand a lot of our friends and colleagues have relied on the eth2 brand for SEO, including block explorers and popular blogs. In particular, Ben Edgington from Consensys mentioned readership of his news website: https://eth2.news/, dropped significantly after trying to remove “Eth2”. This makes total sense today, as the terminology is still valuable before the merge. We agree with Ben on the value it brings towards us as researchers and core developers, and will continue to use the term eth2 in the context of R&D.

One relevant analogy which comes to mind is https and http/2. These were enhancements to the underlying protocols, providing security and scalability to the existing protocols in a way that should be completely invisible to end users. For the purposes of technical discussions it is important to make the distinction, but for end-users of an http/2 system, it’s important that they understand they can use the same browser and search engine stack they’ve always relied on, and even deeper in the stack, people running web servers ultimately should be able to use eg nginx to serve http/1.1 or http/2 or https. End users should not bear this burden.

It is our responsibility as client teams to ensure Ethereum does not become fragmented after the merge, reduce the impact of scammers, and reduce confusion for newcomers. We agree that at this time, it is impossible to stop using the term eth2, and will continue doing so until the merge in certain technical contexts. However, our codebase will be focused on the Ethereum beacon chain, Ethereum validators, and Ethereum proof-of-stake, without adding a 2.0 in the middle.

Hiring

Last but not least, Prysmatic is hiring for a few positions, all remote and full-time: frontend engineer, Go security engineer, and software development engineer in test (SDET). Come join us in working on the future of Ethereum core development! Check out our careers page here, where you can learn more about the roles and how to apply.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Angular and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you.

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address

0x9B984D5a03980D8dc0a24506c968465424c81DbE

Official, Prysmatic Labs ENS Name

prysmatic.eth