Ethereum Sharding Biweekly Development Update #7 — Prysmatic Labs
Latest Research Updates
Sharding and Casper Being Merged Through a Full PoS, Random Beacon Chain
In the original sharding approach, the system was designed to be handled via a smart contract on the Ethereum main chain. This contract would allow for implicit finality via transactions submitting collation headers into it that would then be mined onto a block in the main chain. However, this system is bounded by gas and the current functioning of the EVM 1.0. That is, the number of shards realistically could only grow as much as the sharding manager contract could handle as a load of incoming collation headers.
This system is also more affected by hard-forks and changes occurring in the main Ethereum network. Moreover, the incoming integration of a hybrid Casper PoS system would create a complicated co-existence of two types of validators: namely Casper and sharding validators. Signature verification, in particular, is an extremely expensive operation if done entirely through a smart contract on the main chain, creating inherent bottlenecks in a hybrid system.
Instead, the latest research proposes a sidechain known as a beacon chain that has links to the main chain by containing hashes of canonical main chain blocks within its own block construction.
We will be implementing this system moving forward after our first, proof-of-concept sharding release using our sharding manager contract (our teammate Terence Tsao has already started on a beacon chain implementation in Go!)
Merged Code, Issues, Pull Requests
Simulator and Sync Services
As we are still wrapping up our shardp2p API, we opted for creating a simple simulator service that attaches to a running shard node and simulates incoming requests from other nodes for collation data. When running a proposer, for example, our simulator service sends requests to the running proposer for collation data that has been submitted to the sharding manager contract. The proposer then handles these incoming requests, fetches collation information from its persistent storage, and responds accordingly.
Handling Collation Data Requests Between Nodes
We decided that every node in the shard network should be able to respond to requests for collation bodies coming from shardp2p, not only proposers that signed the original collation headers. In particular, we added a proposer signature as a field in a collation header to the Sharding Manager Contract, which would allow all types of nodes to handle appropriate collation data requests. This functionality is a critical part of our sync process which requests and responds to appropriate p2p messages. We are extending this functionality to allow for windback sync, which involves notaries syncing the previous 25 canonical collation bodies from the shardp2p network.
Validator Registration Contract in Solidity for Beacon Chain
The beacon chain will be implemented independently from the main chain to avoid disruption to the main chain. Only one change to the main chain that is required. A main chain smart contract for validator registration.
In order to become a validator for sharding + casper, a user will deposit 32 ETH to the main chain smart contract. This smart contract is called Validator Registration Contract. The deposit is considered to be burned. As you burn the 32 ETH to participate, the beacon chain will see it and will credit the validator with the validator bond, and the validator can begin to validate. At some point in the future, after a hard fork, the original deposit + interest can be withdrawn back on one of the shards. The Validator Registration contract’s deposit function takes in arguments public key, withdrawal shard ID, withdrawal address and randao commitment. After a validator has deposited 32 ETH it emits a validatorRegistered event.
Merkelize Collations Into Chunk Roots
We finally have been able to merge in #133 thanks to Eli ! The goal of this pull request was for sharding actors to be able to be able to merklize the serialized blobs and derive the chunk root for the collation body. Previously we were able to serialize raw transactions into blobs. Now we wanted to be able to use the serialized collation body and derive the chunk root of the body from it. This also allows us to calculate the Proof of Custody given the collation body, which is used to prove the data availability of a collation when a notary is challenged. The notary will be required to provide a merkle branch of a chunk in the collation body and to provide the merkle root of an altered data tree to prove that the notary actually has that collation.
A major goal for our proof-of-concept release is to increase our code test coverage as much as possible. We value thorough testing as a security requirement as well as an opportunity to increase visibility of the code’s intent. Well written unit tests can serve as a piece of documentation; an example of how to use a given method. These tests also reduce the risk of regression bugs. As we are writing tests to set our codebase up for success, we are already finding and eliminating bugs by way of unit testing. On top of unit testing, we have also chosen to run mutation testing against our code. Coverage is not enough when a line can be removed from the code and no tests fail. Mutation testing is a strategy to mutate each line of code and run the test again, if no tests fail then you might have a problem!
In the example above, we can see that there is a side effect to the Moon method and it should be tested (or removed). Mutation testing helps us realize gaps in our tests and potential places to remove side effect code that should not be there. We use go-mutesting for our mutation testing and we hope to add it to our CI pipeline in the near future.
Current & Upcoming Work
Our Proof-of-Concept Release Product Requirements
Our proof-of-concept release is called the “Ruby” release and will focus around a full implementation of the minimal sharding protocol that will pave way for a greater random beacon chain release soon after. We created a (more than one page) one-pager outlining the following product release requirements based on what we have put together so far:
Sharding Manager Contract
- Our Sharding Manager Contract in solidity implements notary registration, deregistration, addition of collation headers, and voting on collation headers, and retrieval of collation records
- Proposers receive transactions from another process.
- Proposers build complete and valid collations.
- Proposers store collations and can produce them at a later date.
- Proposers submit collation proposals via collation headers -> SMC.
- Join notary pool and stake ETH.
- Be selected on mainchain periods for any given shard.
- Request collation bodies (for committed proposals in the SMC) for a shard during the assigned period.
- Vote on proposals, create proofs of custody, commit to SMC.
- Windback: fetch 25+ previous canonical collations.
- Persist data until challenge period is over
- Generate transactions and broadcast to peers.
- Clients produce collation bodies when requested.
- Clients accept messages for data requests.
- Clients respond to data requests by sending directly to the requestor.
- X proposers running for each shard.
- X notaries running for each shard.
- 1 simulator/observer per shard.
- A mainchain miner with low resource usage (i.e. low difficulty with low cpu cap ~200m).
- Each actor will need its own sidecar mainchain client (non-miner) connected to the miner(s) and other peers.
Interested in Contributing?
We are always looking for devs interested in helping us out! If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).
Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 1 milestone along with a specific project it belongs to (smart contract related tasks, notary node tasks, etc.).
Official, Prysmatic Labs Ether Donation Address
Official, Prysmatic Labs ENS Name