Ethereum Sharding Biweekly Development Update #5 — Prysmatic Labs

Lots of updates to share with everyone this week. From awesome sharding research, to newly merged code, and making steps closer to our first release.

Latest Sharding Research

Cross Links Between Shard Chain and Main Chain

For synchronizing cross shard chain communications, we are researching how to properly link between the shard chain and the beacon chain. In order to accomplish this, a randomly sampled committee will vote to approve a collation in a sharded chain per period and per shard.

As Vitalik wrote, there are two ways create cross links between main shard and shards. On-chain aggregation and off-chain aggregation. For on-chain aggregation, the state of beacon chain will keep track of the randomly sampled committee as validators. Each validator can make one vote casper FFG style, the vote will also contain cross-link of that committee. For off-chain aggregation, every beacon chain block creator will choose one CAS to link the sharded chain to main chain. Off chain aggregation mechanisms have benefits as there is no need for the beacon chain to track of vote counts.

Our Sharding Manager Contract in Solidity

Fixed ETH Deposit Size for Notaries

To recap, a notary must submit a deposit to the Sharding Manager Contract in order to get randomly selected to vote on a block.

A fixed size deposit is good for making the random selection convenient and work well with slashing, as it can always destroy at least a minimum amount of ether. However, a fixed-size deposit does not do well with rewards and penalties. An alternative solution is to design incentive system where rewards and penalties are tracked in a separate variable, and when the final balance when the withdrawal penalties minus rewards reach a threshold, the notary can be voted out. Such a design might ignore an important function which is to reduce the influence of notaries that are offline. In Casper FFG, if more than 1/3 of validators to offline around same time, the deposits will begin to leak quickly. This is called quadratic leak.

Leaderless Random Beacons

In the current research on random beacons, committees are able to generate random numbers if a certain number of participants participate correctly. This is similar to the random beacon used in Dfinity without the use of BLS threshold signatures. The scheme is separated into two separate sections.

In the first section, each participant is committed to a secret key and shares the resulting public key.

In the second section, each participant will use their secret key to deterministically build a polynomial and that polynomial is used to to create n shares (where n is the size of the committee) which can then be encrypted with respect to the public keys and then shared publicly.

Then, in the resolution, all participants are then to reveal their private keys, once the key is revealed anyone can check if the participant committed correctly. We can define the random output as the sum of the private keys for which the participants committed correctly.

One advantage of the leaderless approach is that there is no one leader who has a monopoly on knowledge on the next beacon output and can abort it at will, which was a flaw in the previous RANDAO approach.

A Compendium on Sharding Designs

James Ray from Drops of Diamond, who is implementing sharding for the Parity Rust client, recently came up with this compendium of sharding-related research papers and theory. We highly recommend the readings posted there.

Merged Code, Pull Requests, and Latest Issues

Finished Shard Implementation With Data Availability Checking

We merged a lot of code since the last update! One of the most important PRs involved creating an implementation of a Shard struct with methods for data availability checking and as an abstraction over the backend, key-value store used. The ability to save and fetch collation headers, bodies, and more on the fly is a critical part of our notary or proposer services.

Notaries will be creating shard instances during their assigned periods in order to check for availability and vote on collation headers that will then be finalized in the SMC.

Re-Architecting the Sharding Node (Improving Upon Geth’s Spaghetti Code)

As sharding gives us an opportunity to begin a fresh implementation of Ethereum, we have looked into ways to improve upon Geth’s entry points and architecture of the services attached to a running node. Geth suffers from dependency management hell and spaghetti code where configuration options for certain structs are often spread across multiple files, and it is not entirely clear what interfaces are responsible for different parts of the system.

Moreover, our implementation did not make extensive usage of handling the lifecycle of notaries, proposers, or other services nor did it leverage much of Go’s powerful concurrency primitives. . We began exploring how we can simplify the architecture of our sharding-enabled node while making it extensible to future changes in Issue #122.

We then turned this into Pull Request #127 (now merged), where we revamped our sharding node’s structure to a service oriented approach where we can attach services such as notaries, proposers, and shardp2p managers to a node and have the node be responsible for the lifecycle of these services.

Finished Blob Serialization!

The blob serialization PR #92 was finally merged last week. Currently RLP is used in Ethereum as the main method for serializing objects. The new serialization algorithm differs from RLP in that it separates data in 32 byte chunks with the 1st byte signifying indicator bytes and the other 31 bytes signifying data chunks. The indicator byte contains information on the length of the blob and 3 blob flags. The advantages this serialization algorithm has over RLP are that you can separate questions on availability and validity as everything is a valid encoding of something and that providing Merkle proof of a blob validates that the blob exists in a set of blobs.

Current & Upcoming Work

Improving Our Blob Serialization Algorithm’s Performance

We’ve put together some benchmark tests for our implementation of blob serialization for collation bodies. The numbers from this test show that there is room for improvement. As such, we have created and tagged this issue as a good first issue for contributors. Check it out here.

Investigating P2P

One of our next major features is to implement a new P2P system for a sharded system. The community is considering replacing the current P2P system, DEVP2P, with a library called libp2p. There are a few teams exploring this library for use in Ethereum and tracking in this github issue.

Moving Towards the Ruby Release

Now that we have the SMC and serialization algorithms along with an architecture for a sharding-enabled node, we are building the important foundations for being able to quickly iterate on later research as it occurs.

Our first release will implement the minimal sharding protocol consisting of nodes being able to join a shardp2p network as proposers and notaries that can stake ETH into a sharding manager contract. We plan on having this system as a proof of concept that can spin up a variety of these nodes on a single person’s computer to simulate the functionality and event loops occurring in a sharded network.

We will be organizing our work to make our progress easier to track and align with the ruby release. We will also be communicating the outstanding items required to get this done to everyone following us.

Interested in Contributing?

We are always looking for devs interested in helping us out! If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 1 milestone along with a specific project it belongs to (smart contract related tasks, notary node tasks, etc.).

As always, follow us on Twitter, drop us a line here or on our Gitter chat and let us know what you want to help with — we need all the collaboration we can get! 🎉


Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Like what you read? Give Terence Tsao a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.