Eth 2.0 Dev Update #55 —Medalla Testnet!
Medalla Testnet Launched Successfully
Summarizing the Launch
The entire Ethereum community has been waiting for a stable, “official” multiclient testnet for eth2 for quite some time, and we have finally delivered. On August 4th, the Medalla testnet for phase 0 of eth2 went live, with 5 separate implementations to choose from. At launch, the Ethereum also created the official eth2 launchpad as a great initiative put together by the Ethereum Foundation to help anyone participate in the Medalla multiclient testnet. The launch was broadcast live on Youtube
With high stakes at launch, and client teams rushing to push last minute updates to users, many validators were not perfectly set up at genesis, leading to subpar starting participation. The chain requires at least 66% of validators to be actively validating for the chain to finalize, and we were just short of 57% for the first half hour of the launch. Soon after, more people started coming online, updating their nodes, and participation suddenly jumped to around 80–85%, leading to greater stability in the chain. Although this is not much of a concern during the testnet, the mainnet launch needs to be far more telegraphed, with clients having a stable version a long time before the release.
Overall, Medalla has been an excellent experience, igniting hundreds of people’s excitement for eth2 and getting more people involved. A lot of validators decided to use our project, Prysm, to participate in the testnet and we have been thrilled about the community it now has. We now have over 5000 users in our Discord! If you want to explore the chain’s data, we highly recommend checking the popular block explorer by Bitfly, beaconcha.in, as they have put a lot of work into providing great tools to monitor your validators and their performance.
How to Join the Multiclient Testnet With Prysm
At launch of the testnet, there were 5 separate implementations stakers could choose from, including our project, Prysm. We have put a lot of work into properly documenting how to join the testnet with dedicated instructions here. In summary, during the launchpad process, you can generate your validator keys and submit your deposit, and then import these keys into Prysm to run your validator. Since launch of the testnet, we are now seeing around 65% of the nodes running Prysm (note: this does not count nodes that may be running behind a NAT, only those which have an advertised, public IP address).
Aside from our instructions on how to join Medalla, we also prepared detailed information on all there is to know about wallet and accounts in Prysm, everything about running a validator, and information about how to check if your system is running correctly. If you still haven’t joined the testnet and need some help, join our Discord and our team will be glad to assist you.
New Teammate
Prysmatic Labs has always been a grassroots, community effort. Most us started with a curiosity for eth2, began contributing and asking questions to better understand this multi-year effort. One of our consistent contributors, Radosław Kapka, also known as Radek (https://github.com/rkapka), impressed us with how quickly he’s learned the internals of our code base, submitting non-trivial optimizations to the Prysm client. Radek has many years of experience in software architecture and brings a detail-oriented perspective into building code for eth2. Welcome Radek!
Merged Code, Pull Requests, and Issues
Prysm is up to date with eth2 wallet standards
In the days leading up to the Medalla public testnet launch, we were hard at work on ensuring we align to eth2 wallet and accounts standards for managing validator keys. The testnet launch was a hallmark event for client interoperability, and we needed users to easily transition from the eth2 launchpad into running Prysm. We have completely revamped how Prysm handles validator keys, the wallets we support, and the different customization options available for users. Prysm, by default, supports 3 different kinds of wallets:
- Remote signing wallet An advanced kind of wallet in which validator keys and signing requests are processed by a remote server via gRPC
- non-HD wallet: A simple wallet in which accounts are password protected and imported from an external source. This is the recommended approach if you want to import an account from the Medalla testnet launchpad and you read dedicated instructions here.
- HD wallet: A common type of blockchain wallet which is generated from a english mnemonic, able to create new accounts deterministically.
As a validator, you’re expected to be consistently online to produce blocks and vote on others’ blocks, as this is how you get rewarded for participating in eth2. To do this, your software needs to have instant access to your validating key, also referred to often as a “hot key” or access to a “hot wallet”. Keeping your withdrawal key, or wallet mnemonic far away from your validator client is what will give you optimal security in eth2. If someone were to steal your validating keys, they wouldn’t be able to withdraw your validator’s staked ETH.
The ideal security for an average user participating as a validator is as follows:
- Create a wallet using the official eth2.0-deposit-cli and keep your mnemonic stored offline, safely
- Import only the validating keys you need into your validator client, such as by following the instructions here
For best security in production cloud deployments, it’s best you use a remote signer, as that offers absolute separation of your secret keys and your validator client software. Read more about remote signers here.
Resolving missed attestations reports from users
Users have reported seeing attestation status “missing” while viewing validator attestations in the beacon block explorer. This could happen when there’s a lot of skip blocks in an epoch or this could happen when the validator is offline. Upon further investigations, we did root cause an edge case where Prysm validator unintentionally loses signed committee attestations. Let us explain how that could happen 👇
When the beacon node first starts up, its first job is to find relevant peers in the same committee subnet to gossip committee attestations with. In abnormal cases, this peer discovery process may take more than a few seconds. While the beacon node is still discovering peers, if the validator client requests beacon node to public attestation in the committee subnet where there’s no peer, the beacon node would just broadcast the attestation “into the void”, aka broadcast to no one. Such an issue was cross-checked with the metric `p2p_attestation_subnet_recovered_broadcasts`. We restarted the beacon node and saw `p2p_attestation_subnet_recovered_broadcasts` immediately jumped to 17 while discovering committee subnet peers, that was 17 attestations unanswered! The fix for this is in #6893. With this fix. We are confident that there will no longer be any missed attestation unless the validator client went offline or there’s a heavy chain congestion.
In addition, we have added a RPC debug endpoint to query attestation inclusion slot of a given validator ID and the validator’s assigned committee slot. This is really useful to debug missing attestation and isolate protocol and network failure without suspecting block chain explorer may be displaying data wrong. See #6895 for more info.
Roughtime bug resolved for clock skew
Some of the common errors that users have reported since Medalla genesis are deadline exceeded and the following.
level=error msg=”Could not request attestation to sign at slot” error=”rpc error: code = InvalidArgument desc = invalid request: attestation slot 3558 not within attestation propagation range of 2326 to 2358 (current slot)” prefix=validator pubKey=0x881525ad8f18 slot=3558"
In prysm, we use the roughtime package as a NTP (network time protocol) backup. Why do we use roughtime? NTP responses are insecure and may be subject to MITM attacks. This is especially risky if the host machine only synchronizes its clock every few hours. Roughtime is an awesome project that aims to provide secure time synchronisation. It can resolve clock time within <10s acceptance. While using the roughtime implementation, we suspected there may be a clock skew caused by roughtime in the validator client or beacon node to cause attestation requests that far in advance. We later confirmed our suspicions by the histogram. The clock offset went way up for 1 hour!
To mitigate this issue, the node will now check roughtime more frequently when there’s a large clock offset. At the moment, we only check the clock calibration every hour. #6879 fixes to check every minute instead of every hour. In parallel, #6898 in happening in parallel to make use of roughtime an opt in feature while we evaluate potential issues stemming from erroneous roughtime response.
Fix big issues in local slashing protection and more
A few users have reported local slashing protection cause validator client to panic in #6468. This issue was immediately fixed in #6877, the fix is to prevent a panic in case the history data is nil. We also added better logging in the event where slasher is not up in #6907 also better redial mechanism in #6889 and a race fix in #6871. The local slashing is enabled by default but we also recommend users to run a remote slasher to a more enhanced prevention.
Upcoming Work
Validator API to build a web application for managing your validator client
Modern services running on cloud deployments typically expose a web application for administering them effectively and easily. A highly requested feature for eth2 validator clients has been having a graphical user interface, and we are planning to deliver on this front very soon. Our goal is to have a password-protected local interface you can use to manage your validator accounts, view your validator’s performance, and look at process analytics as needed. We are starting work on the backend and frontend concurrently, and you can take a look at our API design so far here.
Peer scoring and improvements to chain sync
The first phase of a creation of a more sophisticated peer scoring routine has been completed. During that phase we have:
- Updated our peer allow/ban list functionality to be a part of the newly implemented scoring package. Eventually, this scoring information will be used by GossipSub as well, so that evil peers will be excluded from the mesh.
- Created a better protection against slow peers, so that peers returning low number of blocks on average, are used probabilistically less frequently.
- Created a better protection against weak peers, so that peers returning less useful blocks (blocks that cannot be processed), are descored and used probabilistically less frequently.
The next phase will include working with GossipSub protocol by injecting our peer scoring function, and re-arranging the mesh based on relative peers performance. All this will allow for a more smooth synchronization experience in adversary environments. For more information, please, see #6622.
Interested in Contributing?
We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).
Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.
As always, follow us on Twitter or join our Discord server and let us know what you want to help with.
Official, Prysmatic Labs Ether Donation Address
0x9B984D5a03980D8dc0a24506c968465424c81DbE
Official, Prysmatic Labs ENS Name
prysmatic.eth
