Ethereum Sharding Biweekly Development Update #6 — Prysmatic Labs
Latest Research
Registrations, shard count, and shuffling
The validator collateral deposit is fixed at 32 ETH and deposits are done via a one-way burn contract on the legacy chain. A validator can withdraw the deposit in later phase from the beacon chain.
A validator can be in one of the three distinct states:
- Pending Register
- Registered
- Pending Deregister
Registered and pending deregister are both active states and the validators must continue their behavior until they are fully deregistered. The pending states are First-In First-Out (FIFO) queues.
The validator registration work flow is the following:
- Validator deposits 32 ETH to the SMC.
- Validator enters registration queue.
- When the pending register queue reaches the amount of validators required to start one shard, the shard is instantiated.
- The validators in the queue become registered on the new shard.
Registered validators can enter the pending deregister state and deregistration countdown begins. The validator is deregistered when the countdown ends and the number of validators in state pending register is greater than zero. The validator is replaced by the next pending register validator in the FIFO queue.
When count of validators in a pending register state is equal to or greater than the count of validators in registered state, then SMC will increase the shard count by one and move from validators from pending register queue to registered queue. Effectively creating a new shard.
A validator also acts as a proposer and notary. Proposers and notaries are shuffled across shards. Proposers assigned to a shard for 2**19 periods (~30 days) and Notary assigned to shards 2**7 periods long (~10 minutes). Every active validator is active on two shards: once as a proposer and once as a notary. They can possibly be on the same shard, but shard shuffling is random.
The goal is to combine the sharding and proof of stake validators for unity, and every shard has the same number of active validators, proposers and notaries for homogeneity. The ratio of validators to shards is a fixed power of two to simplify design.
Github updates
Design Document Initiative
We’ve started to embrace the practice of writing good design docs. This practice is widely used at Google and other tech companies, but why do they write them?
- Writing helps you think through your ideas.
- Docs facilitate feedback early.
- The design is a blueprint.
These documents record your thinking about a problem and the solutions you are considering. Writing down this information allows the author to really think through ideas before starting work. Team members can understand solutions more effectively and provide their feedback. The document can serve as a blueprint for implementation and assist code reviewers or future readers in understanding why the author has chosen a given solution.
Our process is to write the documents using Google Docs or some other quick collaborative platform, review as a team, and then commit the final doc to our new design doc repository.
Shard Node Design — Finalized and Merged
Our first dive into writing design docs as a team started with refactoring our shard node. As we have started development on a sharding implementation, we have been writing code in an isolated sharding package without much formal design or forward thinking into the framework of a sharding node. The prior work can be accurately described as a discovery and experimentation phase. Now, as we work towards our first major release, we have identified a need to have a formal design for how we bootstrap a sharded node, register different components/services of the application, interact with the mainchain, and run the sharding actor functionality.
If you’d like to read more, the design has been pushed to GitHub.
After the design was finalized, we quickly worked together to implement the refactoring before too many merge conflicts arose. Check out the pull request here.
Our work on this design unblocked much of the feature work and spawned at least a dozen new tasks for contributors to pick up. If you are interested in contributing, there are many new issues with “Good First Issue” and “Beginner” labels.
Serialization Optimization / Benchmarking
Mostly out of curiosity, we created a few benchmark tests on our collation body serialization strategy. It was apparent that there must be some room for improvement. The performance was … not great. So, we put forth an issue for the community and it was almost immediately claimed by Yutaro Mori! About a week later, Yutaro put together a pull request with better benchmarks and huge improvements of about 68% reduction in op time per round trip serialization!
This was our first non-trivial pull request by an outside contributor and we couldn’t be more pleased with their contribution.
New “Good first”Github issues
With the implementation of our new Sharding Node Design , a lot of new issues have been created and we believe that the following few issues are a good first step to start contributing to the project.
Implement Shard Id CLI — This will be so that we can implement a shard Id flag when starting up the sharding client , so that the client know which shard to sync to.
Create Common Testing Interface — This will allow all actors in the sharded system to have a common interface for testing. So that tests do not have to be repeated among different actors
Move Global Config Variables — Move the config variables for sharding to a specific shard object.
Explore SMC with New Elements — Our current implementation in our first release (Ruby release) has us only building the minimum functionality for the minimum sharding protocol. We would like to explore what changes would have to be made to the SMC when taking RANDAO, cross links, dynamic shard count, proposer deposit, better shuffling mechanisms and other new elements into account.
Upcoming Work
Now that we have been wrapping up the functionality of notaries and proposers as part of the minimal sharding protocol, our upcoming work will be focused around the shard p2p and chainsync layers in order to get our proof of concept done.
Our teammate Preston Van Loon put together an initial exploration of a p2p API and interface design in Golang for our shardp2p work in following design doc. We aim to keep as much knowledge of underlying p2p mechanisms internal to the shardp2p subpackage, and will aim to expose a simple API for other services in our system to request and subscribe to messages from peers via an event.Feed interface currently defined in go-ethereum.
Using these initial p2p APIs, we started on a sync design for shardchain data (collation bodies) in the following design doc. We have yet to determine how to handle reorgs as part of the sync process, but this will be part of future discussions and updates.
We are also planning how to containerize our entire proof of concept for the minimal sharding spec in simple commands anyone can run to observe the functionality of the system. Namely, we want to allow users to see transactions being picked up by proposers, notaries syncing and voting on available collations, and the whole system flowing smoothly. We are exploring either Kubernetes or a Docker Compose structure to this problem in the following issue. We determined that an important component of our system is a service called an Observer, that does not participate in notarization or collation proposals, but rather listens on the shardp2p networks and helps with relaying data and other information to peers. This actor can be useful for visualizing how the network is doing on every tick of the system.
Interested in Contributing?
We are always looking for devs interested in helping us out! If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).
Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 1 milestone along with a specific project it belongs to (smart contract related tasks, notary node tasks, etc.).
As always, follow us on Twitter, drop us a line here or on our Gitter chat and let us know what you want to help with — we need all the collaboration we can get 🎉
References
Official, Prysmatic Labs Ether Donation Address
0x9B984D5a03980D8dc0a24506c968465424c81DbE
Official, Prysmatic Labs ENS Name
prysmatic.eth
