Our biweekly updates written by the entire Prysmatic Labs team on the Ethereum 2.0 roadmap
Improving Network Resilience Through Chain Reorgs
Launching a public cluster of beacon nodes that anyone can connect to while maintaining a high degree of network stability and resilience under catastrophes is our top priority. We do not want this to happen:
In the past two weeks, our cloud cluster has become a lot more robust under network latency and restarts of nodes. Even if a large number of validators are offline for a while, as soon as they spin back up, the network is able to successfully continue and penalize those validators for idle behavior. Additionally, when forks occur and nodes rollback their state and head to a common ancestor, we encountered various processing bottlenecks and bugs that are now covered. We use the awesome Jaeger tracing tool to tell us what our bottlenecks are and understand how long every operation in our beacon nodes and validators takes throughout their lifecycle:
Spin Up Any Number of Validators On One Machine
Our new teammate Shay Zluf has been working on a really critical piece of our infrastructure: being able to spin up any number of validators seamlessly on one machine! With a single command, you can attach any number of validators to a running beacon node for them to listen for assignments, propose blocks, and attest to the latest chain head. Every validator represents 32 ETH at stake in the protocol.
Merged Code, Pull Requests, and Issues
Our teammate Ivan Martinez has been working on a series of ETH2.0 benchmarks for our current, unoptimized Prysm client in his README here. A lot of time has been dedicated to understanding some of our processing bottlenecks and thanks to Ivan we’ve taken action to remove duplicated logic, cache important data, and realize how the naive implementation of the spec degrades quickly in performance the more validators are part of the network. The results were analyzed on two criteria: max possible processing conditions (full blocks and all validators performing their responsibilities), and real time conditions from what we expect at launch. We clearly have a lot of work to do and there is a lot of room for optimization all throughout our codebase. If this sort of work interests you, please get in touch with us! There are many interesting and non-trivial methods to have large performance gains in ETH2.0 clients and we are always brainstorming new ideas.
Redesigned Initial Sync for ETH2
We have been critically analyzing how we can facilitate initial sync for ETH2.0 nodes by effectively handling information from peers and ensuring we are caught up the canonical head upon first performing a peer handshake. We request the last finalized state of a node, the current head slot, and all blocks in between. We process blocks sequentially and ensure our resulting state root matches that of the peers who gave us the blocks. If this is incorrect, we blacklist the peer and request from others until we are correctly caught up. Once we exit initial sync, we enter a process called regular sync, which entails listening for block announcements and staying up to date with blocks as they are received. If we receive a block for which we have no parent, we recursively request parents until we aren’t missing any single blocks.
Fixed Critical Attestations Bug
Attestations in ETH2 contain an aggregation bitfield which tracks the index of the validators which voted on a certain beacon block. We had a critical bug in which the length of our bitfield was not padded to account for the length of an attestation committee, which would lead to a major error in our state transition when the length of the bitfield does not match. This took us a long time to debug through a quite painful process, but we knocked one more problem out! Now our attestations look good and all validators are making money correctly.
Caching Validator Committee Shuffling
Recomputing validator committees for each epoch slot is quite an expensive operation, creating a large bottleneck in our testnet the more validators we added. At some point, adding validators would cause the time to process a state transition to take longer than the time required to process blocks. This overflow would completely destroy the system, causing reorgs and nodes to die after a while. Recomputing committees for validators every time is expensive, and that information doesn’t change for a given slot and beacon state, so it’s a prime candidate for caching.
Improving Testnet Visualizations
Our next steps involve really stepping up our visualizations for our real time monitoring of our testnet. Currently, this is what it looks like:
We need to present more information related to validator votes, fork-choice, and peer/network information to give us and ETH2 researchers a better idea of network behavior over time throughout various scenarios, including times of network instability. Aside from this, resolving even more bug fixes and performance optimizations are our biggest priorities before we go live.
Preston & Terence at EDCON Sydney 2019
Our teammates Preston and Terence are at EDCON in Sydney right now joining the larger crew of ETH2.0 teams and sharing ideas for design decisions, optimizations, and improvements to the protocol. Having all these minds in the same room is incredibly productive and has always led to subsequent improvements to the spec and motivates every team to work ever harder towards our shared goal.
New Teammate: Shay Zluf!
We are thrilled to welcome Shay Zluf as our newest core team member at Prysmatic Labs — he brings a wide array of software engineering experience and fit into our team’s flow since day 1. He’s already completed two highly non-trivial features: caching our tree hashing algorithm in simple serialize, and allowing us to run multiple validators clients on a single machine. Welcome Shay!
Interested in Contributing?
We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).
Official, Prysmatic Labs Ether Donation Address
Official, Prysmatic Labs ENS Name