Ethereum Sharding Biweekly Development Update # 10— Prysmatic Labs

Terence Tsao
Aug 7, 2018 · 7 min read

Latest Research

A key piece of Ethereum 2.0 clients is their ability to communicate with each other. Prysm and other clients must have a standard for message formats in order to be interoperable. Prysmatic Labs and other client teams are investigating message serialization formats for peer to peer communication as well as serialization for the consensus mechanism. Consensus requires message data to be serialized in a deterministic manner where ordered fields preserve their order. This requirement is critical for reproducible hashes and therefore consensus.

The ideal solution is a deterministic serialization scheme with an IDL framework and boilerplate code generation. Clients can share the schema definition and generate code in their supported language.There is an ongoing discussion on the ethresearch forums and notes here.

The latest Casper + Sharding 2.1 design spec has reflected changes on the active state fields, crystallized state fields and state transitions. On top of that, the validator cut-off algorithm has been improved to check when there’s not enough validators to fill in a minimally sized committee at every height, the beacon chain node will skip heights. For the validators assigned to each height, the algorithm will further split them up into committees for each shard.

Shuffled validators get split by height then by shard

The fork choice rule itself has been updated to use recursive proximity to justification algorithm for check point selection. After the node has decided on the check points, node applies Casper FFG fork choice rule to favor the chain containing the highest-epoch justified checkpoint. Ethereum foundation team has implemented a network simulator if you want to check it out.

Green: finalized epoch. Yellow: justified epoch. Grey: attestations. Reference: https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ?view

Merged Code, Pull Requests, and Issues

Part of our motivation to move away from geth and into an independent repository owned by Prysmatic Labs containing our different projects would be to allow for the creation of a full beacon node and a sharding client as independent entities. That is, a full beacon node runs a Casper FFG PoS sidechain pegged to the Ethereum PoW mainchain. A sharding client is a separate process that communicates with a beacon chain node via and RPC connection is runs the responsibilities specific to the sharding system including attesters and proposers.

However, our current client/ code implements the Minimal Sharding Protocol created many months ago and is not fully aligned with the responsibilities of the current beacon chain + sharding spec. It still implements sharding through a Sharding Manager Contract on a mainchain node and does not leverage a beacon node for pseudorandomness and attestations.

The purpose of the SMC in the minimal sharding spec and our implementation is to:

  • Handle the submission of collation headers and proofs of custody across shard, period pairs
  • Select committees (attesters) on shards for a certain period
  • Reshuffle committees across shards

These pieces of functionality will be replaced by the Random Beacon Chain and its RANDAO mechanisms + cross-link handling. By leveraging tools such as Google’s gRPC framework for remote process communication and powerful Go mocks, we are currently replacing the SMC with interfaces implemented by a beacon node.

We built a gRPC server in our beacon-chain project as well as a gRPC client in our sharding client implementation and are in the process of refactoring attesters and proposers in the sharding client to match the latest spec while leveraging the beacon chain.

We defined the beacon node RPC services we will expose to the world in this PR and merged it into master.

Beacon Chain Sync involves two steps that are analogous to syncing a traditional Ethereum 1.0 node. Nodes must initially sync to the latest head from other nodes, and then they must listen for new, incoming blocks and apply a fork choice rule in order to accept them as canonical. To accomplish this task, we created a simple simulator of incoming blocks for feature testing that runs within the same process as a beacon node.

For the Beacon Chain initial sync, we start our sync from the block with last finalized slot. Using this we can build a chain till the latest head from the other nodes. Currently when we run the beacon node , it will check if there are any saved blocks if there are it will run a normal sync and listen for blocks to be processed. If there are no saved blocks then it will initiate the initial sync where the node will listen for the latest processed blocks and find out their last finalized slot. Once that is done, they can then query for the other block with higher finalized slots in a sequential manner and add it to the chain. Once the local chain head is the same as the latest chain head in the other nodes, the initial sync is stopped and the normal sync begins. This is being implemented in this PR.

In order for the sync process of a beacon node to function, we require p2p messages specific to the beacon chain. We are currently opting for protobufs despite them being a point of contention in the latest sharding implementation talks. As such, we use the generated code and types from proto messages as canonical types in our Go code, and have implemented beacon p2p topics for p2p subscription through gossipsub. Additionally, our sync service subscribes to p2p messages and handles incoming blocks, crystallized/active state, and more.

We merged this functionality in this PR.

We have implemented state syncing logic within beacon node. We first moved crystallized and active states object to use underlying proto structs. By using generated types from proto mesages, we constructed active state and crystallized state struct. The state sync logic listens to p2p topics for receiving incoming active and crystallized state response via feeds of proto messages through gossipsub. The following tests were implemented to ensure proto structs work and state syncer workflow passes.

Dynasty transition happens when the last two epochs were both justified. We have implemented dynasty transition logic for beacon chain node. When a beacon node sees the last two epochs are justified, it then performs the following operation:

  1. Increment crystallized state’s dynasty count by 1
  2. Go through all the queued validators, check their switch dynasty number and induct them to active validator set
  3. Go through all the active validators, move validators whose balance <=50% of their initial balance to exit validator set
  4. Increment current epoch count by 1
  5. Set current checkpoint to the hash of the previous block
  6. Reset all the balance deltas and FFG voter bitfield

Upcoming Work

The plan for Ethereum 2.0 is to use BLS signature scheme for its efficiency on aggregation. However we are still undecided on the library for Ruby release because the implementation details are subject to change and the pairing implementations are few and far between. The community is leaning towards bilinear pairing BLS12–381 and we are research into the following library. We have created the following issue to track our options. Please comment on the issue with any suggestions or let us know if you are interested to work with us!

Miscellaneous

This past week, teams implementing Ethereum 2.0’s Casper+Sharding spec came together to brainstorm unanswered questions, clarify implementation directions, and determine how to work together as we proceed. Preston Van Loon from our team brought up the topic of P2P message standardization, Danny Ryan from EF led the call, Vitalik and the ETHResearch team came in to clarify items about BLS aggregation and more. This call will become a regular occurrence biweekly on weeks without an Ethereum core devs call.

Check out the detailed notes from the meeting here.

Interested in Contributing?

We are always looking for devs interested in helping us out! If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 1 milestone along with a specific project it belongs to (smart contract related tasks, notary node tasks, etc.).

As always, follow us on Twitter, drop us a line here or on our Gitter chat and let us know what you want to help with — we need all the collaboration we can get 🎉

Official, Prysmatic Labs Ether Donation Address

0x9B984D5a03980D8dc0a24506c968465424c81DbE

Official, Prysmatic Labs ENS Name

prysmatic.eth

Prysmatic Labs

Implementing Ethereum 2.0 - Full Proof of Stake + Sharding

Terence Tsao

Written by

Building Ethereum 2.0 client at Prysmatic Labs @Prylabs

Prysmatic Labs

Implementing Ethereum 2.0 - Full Proof of Stake + Sharding