Recap: Ethereum Sharding Workshop in Taipei
2018–01–28
Vitalik: BTW what do you think about doing a sharding-specific internal kickoff mini-event in Taipei in mid-March, inviting Status, Prysmatic, Pegasys, Parity
Me: Sounds interesting
Turns out it’s not as “mini” as I thought. :D
This was a 3-day internal research workshop hosted by Ethereum Foundation with sharding client implementers and sharding researchers. It’s a great honor and pleasure to have many brilliant Ethereum experts attend this workshop — in Taiwan!
Here, I will recap some of the topics covered during our workshop.
⟠ New specification, new roadmap
If you haven’t seen it, please check the latest sharding phase 1 spec by Ethereum researcher Justin Drake. It’s for public review now.
Also, here is the accompanying infographic we made.
On the day 1 morning, Karl Floersch and I gave a general overview of Ethereum sharding and disclosed some spoilers of the workshop agenda.
The first thing everyone needs to know is — Vitalik and Justin introduced the proposing-collating mechanism: assume you already know that the collation is the block-like structure in the shard chain — the proposer prepares a collation, proposes a proposal (only collation header) with a “bid”, and then publishes the collation body. The eligible collator (i.e., the one who has the right to append a collation) choose the proposal with the highest bid and collate this proposal into collation. In this scheme, the collators only have to deal with the availability of collation bodies and don’t have to execute state transitions.
The whole proposing-collating “game” is more complicated. If you’re interested, please visit https://ethresear.ch.
This new phase 1 spec only introduces consensus of availability— but not any transaction execution and state transition with EVM! The design rationale is that the new roadmap will be more friendly to collator shuffling and more compatible to the Ethereum 2.0 end game.
If you wonder why the collators have to shuffle, here are some explanations with outdated terminologies.
The execution scheme will be introduced in phase 3. Our current research direction is with the new light client protocol: the executors can make claims of the form “I think the state root of block N is H”, then the light client can download the claims because, under honest majority model, the rational executors would provide the correct claims to avoid losing their deposit. Moreover, there’s the improved protocol applied with Truebit-style interactive verification.
One important core of Ethereum sharding mechanism is the Sharding Manager Contract (SMC), a special contract located on the main chain (Ethereum mainnet). Mai-Hsuan Chia introduced the collator shuffling mechanism. This includes: 1) lookahead: the process of providing advance notice to eligible collators before their assigned period; 2) windback: the process of attempting to determine the head by winding back recent collation headers, checking the availability of collation bodies, and applying the fork choice rule.
⟠ Protocol changes for Ethereum 2.0
Ethereum researcher Nicholas Lin and brainbot software developer Jannik Luhn have been working on sharding account abstraction and access list restriction implementations on Py-EVM, and are almost done. These protocol changes will probably be introduced in phase 2.
An important model that Vitalik promoted is the stateless client model. The stateless client only has to store the state root and apply state transitions with witness data, i.e., Merkle branches proofs; therefore, data storage would be largely decreased. However, there are still some optimizations we can perform on the shard chains for reducing witness overheads. Also, during the workshop, Vlad Zamfir suggested that the mechanism of “executing state transition with the witness data” does not have to be not limited to “stateless” and could be conceptually called a “self-authenticating blocks” scheme — the client could choose to store state optionally under this scheme.
⟠ Rent is the new black
IC3 researcher Phil Daian introduced the concept of rent. Phil deems it’s unfair that in the current system, the blockchain miners decide inclusion, but the miners are not the only ones bearing the cost. To solve this situation, we need to develop a new blockchain resource economic model of charging with “time value” of storage. However, it’s worthwhile to mention that implementing this idea in the DApps will worsen UX for developers.
There is an inspiring discussion about the rent topic in the following Taipei Ethereum Meetup.
⟠ P2P networking
Felix Lange and Péter Szilágyi gave a presentation on geth sync mechanism and challenges. For sharding, collator shuffling requires the ability to jump between the P2P networks quickly, but the current Ethereum P2P network doesn’t support this. Thus, a new sharding network topology will be an important topic for implementation.
⟠ What’s-a what’s-a what’s-a what’s-a WASM!
eWASM team introduced the eWASM architecture, and discussed WASM integration into the current client implementation with Parity WASM. We also discussed what’s required for sharding eWASM.
⟠ Binary consensus and sharding
Vlad’s sharding mechanism (a.k.a. Vlarding) is based on the binary consensus system with complete mathematical definitions. The whole CBC Casper + Sharding lecture was presented at EthCC.
⟠ Mechanism design
1) Execution-minimization and state-minimization
Justin has plenty of optimization and minimization ideas for Ethereum 2.0. In this section, Justin explained and compared the tradeoffs between different mechanisms, and why we should minimize execution and state.
2) Cross everything
Cross-contract and cross-shard communication require more delicate procedures. In an asynchronous communication model, extra security guarantees are necessary to make sure the message from one shard gets included in another shard within the time threshold.
And, in synchronous communication models, the problem is how to make cross-shard atomic synchronous transactions. Atomic transactions are essential as otherwise sending Ether (or making calls) from one shard to another may be dangerous — it could cause Ether to disappear or to duplicate. In Vlad’s sharding paradigm, the fork-choice rule enforces atomicity of merge blocks.
Another method is to generalize the cross-shard contract locking scheme and the yanking scheme that Vitalik and Piper Merriam just came up with during the workshop!
3) Scalable data availability checking
Vitalik explained the data availability problem in P2P networks. Simple fraud proofs are not a solution since not publishing data is not a uniquely attributable fault. With honest minority assumption, we can use erasure codes to encode the block data into “chunks” and merklized them. In this way, the light client can use this extended data and fraud proofs to verify data availability probabilistically.
Another interesting idea is the concept of Proof-of-Custody introduced by Vlad which allows you to prove that you have custody of an entire file. Furthermore, Justin discussed a DFinity-style scheme using BLS signatures and an honest majority assumption to build availability proofs.
4) Security models
Vitalik and Vlad have been investigating this topic for years. During this part of the workshop, we discussed security goals (safety and liveness), different security models, and evaluated sharding.
In phase 1, we have security under the honest majority and the uncoordinated majority model. However, the proposing-collating scheme creates a perfect environment for bribers. How to resist bribing attacks in the sharding protocol is an issue we need to figure out in phase 5 — tight coupling.
Ethereum 2.0 end game
1) Full Casper integration
Vitalik foresaw that tightly coupled sharding would be integrated with full Casper.
2) Forkful or Fork-free
Regarding post-tight-coupling sharding, Justin has a proposal for a fork-free scheme while Vlad would say: Nah, we need fork! 🍴
Resources
🦄 Resources will be updated here.
🦄 Also, Vitalik updated the Sharding FAQ after the workshop.
Acknowledgments
Thank you to the Ethereum Taiwan team: Chih-Cheng Liang for preparing all the meals and sharing burdens, Mai-Hsuan Chia for contacting GIS TAIPEI TECH Convention Center and all the venue staff, Nicholas Lin for the logistics, and Dr. Chang-Wu Chen for being MC. Also, thanks to the contributors Yu-Wei Lin and Adrian Liaw for your help.
Thank you to our ED Aya Miyaguchi, researcher Jon Choi, and admin assistant Toya Budunggud for your helpful advice on how to organize the event. Thank you to our advisor Albert Ni for helping review this post.
Thank you to the Ethereum Foundation research team, especially Justin Drake, for the excellent sharding research work that brightened up this workshop, Karl Floersch for being my Amy Poehler, Nate Rush for giving constructive input in the workshop and helping to review this report, and Vlad Zamfir for the inspiring sci-vlarding wubba lubba dub dub!
Thank you to my favorite guests (yep, I’m not impartial): Piper Merriam (Ethereum Foundation Python implementation team, Trinity), Péter Szilágyi and Felix Lange (Ethereum Foundation Go implementation team, geth). One of the best things in this workshop was the communication between research and implementation.
Thank you to the eWASM team for the remote presentation. Sorry for the rushed plan — you are awesome!
Thank you all the attendees from different organizations who came to Taiwan from a great distance for the same goal — scaling Ethereum: Status Research & Development GmbH, ConsenSys, Parity Technologies, Prysmatic Labs, Drops of Diamond, brainbot technologies AG, Web3 Foundation, IC3, Cornell Tech, Cornell University, Singapore University of Technology and Design, L4, Counterfactual, Barcelona Supercomputing Center, and AMIS. Thank you for spending your valuable time, participating and engaging this workshop, and devoting yourself to Ethereum. You built this.
Big thanks to Vitalik Buterin for being a genius, suggesting this great idea, and all the support!
By the way, Ethereum research team is looking for more talented devs to join our sharding implementation work. 🦄️
And thank you for reading!
