Ethereum 2.0 Development Update #36 — Prysmatic Labs
Our biweekly updates written by the entire Prysmatic Labs team on the Ethereum Serenity roadmap.
🎉Testnet is online🎉
We have restarted our test net for everyone to experience staking and becoming a validator. This test net includes beacon chain spec v0.8.4, various performance improvements, faster BLS paring library, new syncing strategies and more RPC end point support. Please give it a try and Hop on our discord and open issue for feedback: https://prylabs.net/participate
Devcon
Prysmatic Goes to Devcon
Some of our team members will be attending Devcon V, the largest Ethereum conference in the world this upcoming week in Osaka, Japan. Our very own Terence Tsao will be giving a lightning talk summarizing our latest work on Prysm and what to expect from our team moving forward as we get closer to the mainnet release of the beacon chain. We’ll have Prysm stickers to give out as well — so please come chat with us at the conference!
Merged Code, Pull Requests, and Issues
Massive BLS Improvements from New Contributor
Given the lack of BLS cryptography options in Go, we were quite limited in terms of performance over the past few months. In terms of compatibility with our tooling and build system, there just wasn’t a good option out there, so we opted to keep things simple and use a prototypal Go implementation called https://github.com/phoreproject/bls which met our basic needs of passing spec tests, but would severely limit us from running a larger number of validators in our public testnet. We were happy it worked, but were in desperate need of a solution. Recently, a new open source contributor, oniki (Onur Kılıç), popped up in our discord channel telling us he’s been working on a BLS improvement using assembly bindings. On a 2.7 GHz i5 machine, the benchmark for the BLS pairing was reduced by an entire order of magnitude from our previous one!
When running it in production, BLS is now around 20% of our bottlenecks, compared to 99.99% before.
Onur’s work has been incredible so far and we are looking forward to seeing more innovation come from him.
Large Prysm Performance Improvements
Given the lack of BLS cryptography options in Go, we were quite limited in terms of performance over the past few months. Due to this, most of our benchmarks and profiling would be 99% filled up by BLS as an outlier, preventing us from seeing deeper problems with our runtime that could be radically improved. We added a feature flag in development mode that would make all instances of BLS no-ops, so we could see what else was causing serious bottlenecks in our nodes. Due to this, we opened up a pull request to radically revamp our performance:
A ton of our bottlenecks came from our DB writes, which were usually not batched, causing most writes to be sequential and slow due to mutexes acquiring lock permissions to write to disk in order. Our backend, the key-value store project BoltDB, is nicely optimized for batch writes that we were not fully using. After fixing this, our biggest bottlenecks, which would be in updating validator latest votes + saving attestations, were reduced by nearly 30%!
Round Robin Network Sync Complete
Round Robin network syncing has been released and is working well. When clients sync with the Prysm testnet, they’ll query all of their peers in order with a subset of the request. This algorithm is effective at minimizing the trust and burden on any single peer. Give it a try by following the instructions on our test net website at https://prylabs.net!
Ethereum APIs V1 Alpha Complete
For quite a while now, we’ve been working on creating a solid, well-tested, well-documented API suite for the Ethereum beacon chain under the EthereumAPIs repository here. We have fully implemented v1alpha1 into Prysm and we have exposed a public API endpoint at https://api.prylabs.network for everyone to retrieve data from our live, public testnet. We used the awesome Swagger frontend to create a nice interface to browse our API endpoints and even try them out with real inputs and real responses.
A sample query for the current beacon chain head block from our live public testnet — try it out!
https://api.prylabs.network/eth/v1alpha1/beacon/chainhead
Archival Node Functionality Complete
It’s really important for the purpose of block explorers and other data gathering use cases of Ethereum to run archival nodes. An archival node not only stores data required for consensus, but also historical information such as validator balances, blocks, attestations, and more since the beginning of time. We completed the archival node functionality and plugged that into RPC API, allowing anyone to request historical information, even genesis information, if the beacon node is running with an ` — archive` flag.
Critical Prysm Bug Fixes Roundup
In anticipation for our testnet relaunch, we have been trying to improve Prysm’s runtime performance with a larger number of validators. However, this has lead to some critical runtime bugs popping up which lead our node to be dysfunctional. The majority of these bugs have been due to attestation aggregation; with a large number of attestations being sent over the wire, processing them and aggregating them efficiently is a non-trivial problem. Currently we store attestations received from the wire in an attestation pool, and then in there, aggregate them. However, aggregating them correctly can be an issue especially as we process multiple attestations concurrently. We had some of our attestations being mutated in the process of aggregating, which lead to invalid signature verifications. This was resolved here.
Standardize Caching
Caching is one of the most important runtime component. As the spec aims for readability, it’s not meant to be performant. Implementing directly from spec may result in slow run time.As a running node, it’s important to ensure runtime computation can be done as efficiently as possible. Starting from the first version of the spec, Prysm has multiple variations of run-time caches aim to optimize active balance, shuffling seed, start shard, total validator balances, active balances, and shuffled indices. One of our Q4 goals is to standardize the caching scheme and reduce disparity between all variations. The latest design is to precompute shuffling as a full list and prepare all the committees for easy access. The list does not change throughout the epoch and can be known one epoch in advance. You can track our progress here.
Upcoming Work
Shorter Duration Testnets, Stress Testing
Ethereum Serenity is a synchronous protocol, meaning time plays an important role in consensus. Validators are assigned specific timestamps in which they are allowed to create a block or vote on a block, and must perform their responsibilities accordingly. Currently, this is known as the SECONDS_PER_SLOT configuration parameter of the protocol, set to 6 seconds per slot based on performance assumptions clients should comply by for mainnet. However, just because this value is set to 6 seconds doesn’t mean we should stop there. A big part of being production ready means we can successfully run testnets with 1 second slots! To do this, a lot of things needed to be optimized, but if done correctly, we can have a lot of confidence 6 seconds will be more than enough to handle everything validators need to do within that assigned time frame. We have opened up a tracking issue for this task here.
Improving Sync Performance on Public Testnet
Next big task for networking is to find ways to minimize processing times for syncing with the running testnet. At the current sync rate of 20 blocks per second, each day adds 12 minutes of sync time to a new client. By design, clients want to sync the last weak subjectivity period of 9 months. We’re looking at a 54 hour syncing time at the current rate, for the entire weak subjectivity period, without significant improvements to the sync processing pipeline. Ideally, this is something we aim to do in under 12 hours, but at least ETH2 has an upper bounds on sync times whereas the Ethereum 1 chain sync grows indefinitely.
Benchmarking Worst Case Conditions
While our testnet handles the amount of validators using the network with no issues, in order to be production-ready, we need to be prepared for production-level conditions. Most test cases written for the client usually involve around 100 validators to make sure the tests don’t take long to run, but we need to start benchmarking worst case conditions to make sure we’re able to handle them when the time comes.
One important tool that can be helpful for benchmarking is a simple way to generate valid and configurable blocks so we can test various worst case conditions easily. Once we can reliably generate valid data to test with, we’re intending to perform benchmarks of a state transition with 65536 validators attesting through 128 and 256 unique attestations. Using such a large set of validators, the bottlenecks in our client will be in clear sight and we can start ensuring we are prepared for mainnet’s worst case scenarios.
Fuzzing Tests
Fuzz testing is a method used to detect edge cases that were not tested in the codebase itself by calling methods with random values. It is mainly beneficial for detecting serialize/unserialize bugs and detection of wrong object formats that may cause our project to panic. We have started designing our fuzzing tests approach while considering making part of this effort common to all eth2.0 clients.
Interested in Contributing?
We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).
Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.
As always, follow us on Twitter or join our Discord server and let us know what you want to help with.
Official, Prysmatic Labs Ether Donation Address
0x9B984D5a03980D8dc0a24506c968465424c81DbE
Official, Prysmatic Labs ENS Name
prysmatic.eth
