On the Origin of Zilliqa

Published in

On The Origin of Smart Contract Platforms

19 min readOct 23, 2018

This article is part of the ongoing “Origin” series that tracks the emergence and evolution of projects in the cryptocurrency-based smart contract ecosystem. Today we’ll look at Zilliqa and the challenge it poses to today’s dominant platform, Ethereum.

Zilliqa is a smart contract platform built on a sharded blockchain that has clocked in at thousands of transactions per second (TPS). It includes a new functional language for developing smart contracts that facilitates automated formal verification of security properties. It is intended to function as an open, public blockchain for running smart contracts and thus is a direct competitor to Ethereum.

The project’s name is apparently a play on “silica” that means “Silicon for the high-throughput consensus computer.” The company was founded by a group of researchers from the National University of Singapore including Xinshu Dong (CEO), Yaoqi Jia (Head of Technology), Amrit Kumar (Head of Research), and Prateek Saxena (Chief Scientific Advisor), and others. Saxena had worked with Loi Luu (founder and CEO of Kyber Network and advisor to Zilliqa) and other researchers at NUS on the ELASTICO protocol for sharding permissionless blockchains. The ELASTICO paper would serve as the basis for Zilliqa’s sharding technology.

Zilliqa incorporated in Singaport on June 1, 2017. By September 1 the team had launched its first testnet (v0.1), and on October 12 announced that the testnet (v0.5) had achieved a peak throughput of 2,488 transactions/sec. On December 1 the testnet (v1.0) was opened to the public, and in January 2018 the code was open sourced. Zilliqa raised about 20 million USD in private funding in the Fall of 2017, and then raised another 2 million USD during the one week (December 27, 2017 to January 4, 2018) ICO for its ZIL token.

Zilliqa directed some of its funding toward creating the #BuildonZIL Ecosystem Grant Programme to support teams and developers building on its platform. On August 14, 2018 the company announced its first round of grants to fund a wide range of projects to develop wallets, libraries, developer tools, and applications. Zilliqa is also the platform being used in Project Proton, a joint initiative to assess the viability of blockchain technology to bring transparency and accountability to programmatic advertising. The launch of Project Proton’s first phase was recently announced. Presumably, a number of the #BuildonZIL projects will launch with Zilliqa’s mainnet release, which should occur sometime in the winter of 2018-2019.

Will 2019 be the year Zilliqa’s high throughput smart contract platform makes a big disruptive splash? Or will it be the year that scaling advances in Ethereum magnify its network effects and move it into an even more dominant position? Will another high profile Solidity hack create a community crisis sending smart contract developers heading towards safer platforms? No one can know for sure, but an in-depth look at the technology being developed on these two platforms should give us some clues. This article will take a closer look at the scalability and smart contract security of the Ethereum and Zilliqa projects.

Scalability

The most recent benchmarks run on Zilliqa’s (v1.0) testnet demonstrated a throughput of 2000–3000 TPS. Zilliqa achieves this two orders of magnitude increase over Ethereum through sharding, a base layer scaling solution that increases throughput by allowing transactions to be processed in parallel. Without sharding, all mining nodes in the network must validate every transaction added to the blockchain, creating a throughput bottleneck. With sharding, the mining nodes are partitioned into groups, called shards, that are only responsible for validating the subset of transactions that have been allocated to their shard. This allows transactions on different shards to be processed in parallel, increasing transaction throughput.

In theory, demand for higher transaction throughput can be met by simply adding new shards and increasing parallelism. This ability to increase capacity with demand is known as linear scaling. In practice, linear scaling can be realized for simple transfers of funds from account to account, but difficult to achieve for smart contract transactions that need to manage state across shards because of the parallel slowdown that can occur due to the overhead of cross-shard communication and state synchronization.

The plan for both Zilliqa and Ethereum is to first implement basic transaction sharding (i.e. simple payments within shards) and then tackle the much more difficult problem of state sharding (i.e. arbitrary state changes that must be kept consistent across shards) in later phases. The remainder of this section will be broken into two parts, with the first covering the latest implementations of transaction sharding in both projects and the second discussing the research and approach each is taking to address state sharding.

Transaction Sharding

In Zilliqa’s first, payment only implementation, each transaction is assigned to a shard based on the rightmost bits of the sending account’s address. For example, in a network of 16 shards numbered (in hex) 0…f, the last hex digit of the sender’s account would identify the shard a transaction is assigned to.

Assigning all transactions originating from the same account to a single shard is the simplest and most efficient way of preventing double spends in a sharded blockchain. All shards can run in parallel since miners don’t have to worry about a potentially conflicting spend being mined on another shard. An account can receive funds on any shard, but can only spend them on their assigned shard and only after the receiving transaction has been immutably hashed into a final block on the main chain. Below is a diagram illustrating how this process works.

Zilliqa’s sharding implementation consists of two consensus layers. At the bottom layer, miners in the individual shards create micro blocks of the transactions assigned to their shard. The header and signature of each micro block is sent to a Directory Services (DS) committee at the top layer, which aggregates the headers into final blocks that make up the global ledger for the system.

For simple transaction sharding the DS committee does not process the individual transactions, just the micro block headers. Doing otherwise would eliminate the benefits of parallel processing and defeat the whole purpose of sharding. The DS committee assumes that the lower layer consensus process has correctly validated all transactions in its micro blocks. Thus secure shards are key to overall system security.

To reach consensus on blocks, Zilliqa employs a Byzcoin-inspired, optimized version of the Practical Byzantine Fault Tolerance (PBFT) protocol that achieves about 415 TPS per shard. The primary optimization is to replace message authentication codes with aggregated Schnorr signatures. This greatly reduces the number of messages required to reach consensus (from quadratic to linear in the number of nodes) and makes it practical to implement shards with hundreds of miners.

A nice property of PBFT that greatly simplifies the sharding implementation is instant finality. Once consensus on a micro block is achieved, the block can be immediately processed by the DS committee without having to wait for confirmations since chain reorganization in the shard is impossible.

The safety and liveness of the PBFT protocol requires more than 2/3 honest nodes in each shard and some bounds on message delivery time (i.e. partial synchrony of the network). Achieving this in an open, public network where anyone can participate in consensus requires some defense against Sybil attacks, in which an attacker can subvert consensus by creating a large number of nodes.

To deter Sybil attacks, Zilliqa requires each node to generate a proof of work (PoW) before it can participate in either the DS or shard consensus process. PoW is not used to qualify each block produced (as in Ethereum), but only to qualify nodes to participate in a consensus process. Once a node has qualified for participation it can generate blocks as fast as possible without having to perform PoW again until it needs to re-qualify (much like the Bitcoin-NG algorithm used in æternity). This allows the protocol to achieve high transaction throughput with a low energy footprint.

The PoW algorithm used by Zilliqa is Ethash, Ethereum’s PoW algorithm. Ethash is designed to be ASIC-resistant; however, Bitmain is currently shipping an Ethash ASIC miner that could be used to mine Zilliqa. Whereas the first generation miners were marginally more efficient than GPU rigs, the next generation will likely deliver significant gains in energy efficiency that could lead to centralization of mining power. Zilliqa has an opportunity to avoid this issue (at least for a while) by choosing a different memory hard algorithm for which no ASIC currently exists.

Ethereum is slowly evolving toward a sharded version of its blockchain. The latest Ethereum 2.0 spec describes a two-tiered system that has architectural similarities to Zilliqa’s (see the figure below).

Much like Zilliqa, Ethereum has a number of shards on which blocks of transactions are produced in parallel by committees of validators randomly assigned to the shard. Signed hashes of the shard blocks are mined into Beacon blocks on the Proof of Stake (PoS) Beacon Chain. The Beacon Chain functions much like a combination of Zilliqa’s DS committee and Main Chain, aggregating shard blocks as well as managing the assignment, rotation, and financial incentives of validators. Ethereum’s PoW mined main chain (Legacy Chain) plays a very small role, accepting validator deposits and transferring them to the Beacon Chain. The “Legacy Chain” moniker now being used reinforces that Ethereum is following Vitalik’s proposal for moving to full PoS in which the PoS Beacon Chain ultimately replaces the PoW Legacy Chain as Ethereum’s main chain.

The validators assigned to a shard can act as Proposers, who produce shard blocks, or Attesters, who vote for blocks. Each Attester vote is signed and serves as an attestation to the existence and validity of a shard block and a corresponding block on the Beacon Chain. Attestations from a quorum of validators create crosslinks, which finalize checkpoints on the Beacon Chain and the individual shard chains. These checkpoints can be used to enable safe cross-shard communication (see State Sharding below).

Consensus and finalization of blocks is achieved using a modified version of Casper FFG (described here and here) that is (accountably) safe and (plausibly) live provided there is an honest majority of validators. If an attacker controls 1/3 to 1/2 of validator stake then safety depends on partial synchrony assumptions and above 1/2 the attacker can revert recent (non-checkpointed) blocks. Although it assumes no bounds on message transmission delay, the Ethereum 2.0 spec requires each validator’s clock to be synchronized with all others in order to process blocks in fixed 8 or 16 second slots.

The throughput of Ethereum 2.0 with 1024 shards has been estimated at 13,410 TPS. This is a theoretical number that doesn’t take into account that throughput slows down when transactions must synchronize across shards, and the number of transactions requiring cross-shard synchronization increases with the number of shards. The next section looks at why this cross-shard synchronization is a necessary and challenging aspect of implementing smart contracts on sharded architectures.

State Sharding

Throughput estimates and benchmarked measurements can show very high theoretical transaction rates when they are based on simple payment transactions, which are highly parallelizable. However, when smart contracts are involved, practical throughput is significantly lower because parallelization is not as simple as just assigning the transaction to a shard based on the sending account’s address.

Consider a simple contract that allows users to reserve a seat on a train by sending a payment to the contract. If Alice’s account address and Bob’s account address map to different shards and each issues a transaction to reserve a seat, the two transactions cannot run independently in parallel on different shards because the state of the contract, i.e. the number of reserved/available seats, creates a dependency among the transactions. If there is only one seat left, and Alice’s transaction succeeds in reserving it, then Bob’s transaction must fail because there are no seats left. One might consider assigning such transactions to a single shard based on the contract address (and somehow deal with potential double spends from Alice and Bob, for example by requiring them to generate and pay with addresses that map to the contract’s shard) but this strategy breaks down whenever two or more contracts (whose addresses don’t coincidentally map to the same shard) are involved.

A wide variety of transactions do more than one thing involving more than one contract and/or more than one user’s account. For example, a simple ERC20 token exchange will involve updating state on two separate ERC20 contracts for both buyer and seller. The balances to be updated map to buyer and seller accounts whose addresses, and thus assigned shards, aren’t easily changed, and the contracts are deployed by independent organizations with little incentive to coordinate their contract addresses.

Transactions that span multiple shards are unavoidable. Thus a viable sharding solution must provide a secure and efficient cross-shard communication mechanism. The Ethereum community has developed such a mechanism that involves breaking the transaction into two transactions that run sequentially on different shards. When the first transaction is mined on its shard a receipt is created that can be used by the second transaction to prove to miners on its shard that the first transaction executed. Once the second transaction is mined, the receipt is considered “spent” so no other transaction can use it.

This split transaction mechanism is referred to in the Ethereum community as asynchronous cross-shard communication because the mining of each transaction into a block happens independently. There is no bound on the amount of time it will take for the second transaction to be mined once the first transaction has been mined or even any guarantee that it ever will. This presents a practical issue for transactions that make dependent state changes, such as an ERC20 exchange transaction. If Alice and Bob are swapping tokens it’s a poor user experience for Alice to have the tokens in her account for an unbounded period of time before Bob has his, and unacceptable for Alice to receive Bob’s tokens without Bob ever receiving hers.

One proposal for addressing this atomicity issue is contract yanking, wherein one ERC20 contract’s state is frozen and temporarily “yanked” over to the shard where the other ERC20 contract resides. There both contract states are updated atomically and the yanked contract’s updated state is returned to its shard and unfrozen. This solution also leverages receipts to move the contracts across shards and thus has some latency issues that could keep a contract frozen for an impractical amount of time.

The Holy-Grail solution is what the Ethereum community calls synchronous cross-shard communication. Ideally transactions spanning multiple shards would get executed at the same block height across all shards involved. This is especially challenging to accomplish on Ethereum, since any one of the involved shards could undergo a chain reorganization (Zilliqa’s PBFT consensus does not fork). A couple of proposals (involving state execution engines and chain fibers) have surfaced but a working solution still appears to be a long ways off.

The state of sharding development in Ethereum appears to be somewhere between phase 0 and phase 1 on the sharding roadmap. However, at the EthBerlin hackathon in early September a team of developers, including Casper Researcher Vlad Zamfir, created a milestone proof of concept for asynchronous cross-shard communication. According to Zamfir, the implementation “prevents finalization of cross-shard atomicity failure, so it will never be that a send is finalized and a not received is finalized.” Whereas finalized checkpoints enable this safety guarantee, relying on them introduces an additional latency between the time the send and receive are finalized on their respective shards.

The instant finality of blocks in Zilliqa prevents forking and provides an opportunity to significantly reduce and tightly bound the latency of safe cross-shard communication. Zilliqa’s approach to the problem is described in this blog post, which is also an excellent exposition on the challenges of implementing smart contracts on a sharded blockchain. Zilliqa’s proposed solution categorizes transactions into three types: (I) simple payments between two users, (II) simple transactions where a single user interacts with a single smart contract that does not transfer funds to another user nor call another smart contract, and (III) all other transactions involving smart contracts. All category I and the subset of category II transactions where the user’s address and smart contract address map to the same shard can be validated on the shard; all other Category II and III transactions must be processed by the DS committee on the main chain.

As the proportion of transactions that must be processed serially by the DS committee increases, parallelism decreases and the overall transaction throughput approaches that of the main chain. Metaphorically, if a six lane highway requires certain types of vehicles to stay in a single lane, then the extent to which it experiences a traffic jam will depend on how many vehicles are assigned to that lane. Zilliqa’s analysis estimating the percentage of serial vs parallel transactions omits all Category III transactions involving multiple contracts or multiple users, 100% of which would need to be processed serially on the main chain. I believe that many typical transactions will fall under this category and omitting them from the analysis paints an overly optimistic picture of the throughput Zilliqa will achieve in practice.

It will be interesting to see what TPS numbers Zilliqa achieves when running real smart contract transactions on an operational mainnet. If things go according to Zilliqa’s plan, this will occur before Ethereum smart contracts are running on shards, so there could be some period of time during which Zilliqa’s platform offers a higher throughput base layer than Ethereum. However, scalability — the degree to which throughput can be increased by adding new shards — will depend on the implementation of cross-shard communication mechanisms that don’t become major bottlenecks as the number of shards increases. Zilliqa will likely need some innovative mechanism to overcome its current cross-shard communication bottleneck and deliver on its promise of linear scaling. If it does, the significant reduction in cross-shard latency that instant finality brings can make Zilliqa a better target than Ethereum for certain classes of latency-sensitive applications.

Smart Contract Security

As a first generation smart contracts platform, Ethereum has revealed the practical challenges in developing secure smart contracts. Even the most experienced developers following the best security practices have deployed contracts with exploitable bugs that resulted in loss of users’ funds. In the wake of these hacks debate has ensued between those advocating for some governance action (often a hard fork) to restore lost funds and those who are opposed to arbitrary bail outs and/or wishing to prevent a community split. Indeed, a community split over this very issue is what led to the creation of the Ethereum Classic fork.

Ethereum is still in the early stages of defining government processes to address such issues, and until it does the community’s general (and perhaps majority) opposition to bailouts suggests that lost funds will not restored. In the absence of a governance process the community must take a preventive, rather than reactive approach, to securing users’ funds.

Several projects within the Ethereum community are addressing this issue by developing code analysis tools and safer programming languages including the proprietary Axlang and SolidityX languages and the open source Vyper language. Vyper is designed to improve security by making the code simpler (easier to audit) and less likely to mislead developers and users about its behavior. As such it is non-Turing complete, decidable, strongly typed, and intentionally omits several features that can result in surprising and thus vulnerable code. Vyper is currently in beta, and not yet widely used to develop smart contracts on the Ethereum mainnet.

Zilliqa’s approach to smart contract security was likely informed by the work of its Lead Language Designer, Ilya Sergey, and Chief Scientific Advisor, Prateek Saxena, on MAIAN. MAIAN is a static code analysis tool that finds bugs that allow EVM smart contracts to lock funds indefinitely, leak them to arbitrary users, or be killed by unauthorized users. To prevent those types of bugs, Sergey and other researchers from Zilliqa/NUS developed Scilla, a functional, strongly typed, non-Turing complete programming language that is being used to develop safer smart contracts on the Zilliqa platform.

Scilla is designed to facilitate formal verification of smart contract programs. Formal verification doesn’t guarantee that a smart contract is free of bugs, but it can be used to develop proofs that a contract cannot violate certain specified security properties. Such proofs can give a contract’s users confidence that their funds cannot be stolen or frozen due to vulnerabilities in the code.

To facilitate construction of proofs, Scilla makes a few key design tradeoffs. It is designed to be an intermediate-level language, onto which more expressive and developer friendly languages (including Solidity) can be compiled. It follows functional programming paradigm by clearly separating purely functional computation from state-modifying actions, which are always made explicit in the code. Finally, it requires all contracts to be structured as communicating automata. In this structure programs are defined as a set of state transitions that are triggered by incoming messages with all interactions between contracts occurring at well-defined boundaries within the code.

Scilla smart contracts are written as a set of transitions, which are like functions in most other languages except that all calls to other contracts can only be executed as the transition’s final step. This restriction allows formal verification tools to assert how the state of the contract will change during a single transition without having to worry about an external call causing arbitrary state changes (e.g. by calling back into the calling contract). This enables the detection and prevention of a certain class of reentrancy vulnerabilities, including the one exploited in the DAO hack.

Most smart contracts can (and should) be written in this so-called tail-call programming style. The Scilla whitepaper originally described a method of using continuations — blocks of code that are invoked by the execution environment when an external call completes — to translate non-tail-call contract code from existing contracts into Scilla’s communicating automata form. However, the Zilliqa team has informed me that continuations are no longer part of the language, so existing contracts would need to be rewritten in tail call style before being compiled to Scilla.

It’s still possible to develop reentrant code in Scilla, but vulnerabilities due to reentrancy are less likely because tail call style makes the developer (and verifier) explicitly aware of what state changes are happening at any given point in the program. Looping constructs are implemented via recursive function definitions so their termination can be proved statically. To address recursive loops that could exhaust resources Zilliqa implements structural recursion on lists and requires transactions to pay gas like Ethereum.

Although Scilla is still under development and tooling around it needs to be developed, its focus on simplicity and formal verification will make it a solid building block upon which safer smart contracts can be built. While Ethereum is also moving in this direction, the evolution has been slow and thousands of contracts deployed on the platform remain vulnerable. As we’ve seen with the DAO and Parity hacks, high profile exploits can be highly disruptive to the community, though surviving them makes the platform stronger and safer going forward. To date, one reason Ethereum has survived these events is the lack of a safer alternative. At some point, when high level languages and tooling built on Scilla exist, Zilliqa may be able to provide that alternative.

Conclusion

Zilliqa is a project that grew out of research at NUS and that research continues as the project addresses some of the major challenges that Ethereum faces concerning scalability and smart contract security.

Both projects are addressing base layer scalability with sharding and doing so with roughly similar architectures and road maps. The easy problem, transaction sharding, has been solved and one can expect implementations from both projects to be deployed in 2019. Zilliqa has an advantage in starting from scratch without the burden of supporting a PoW main chain holding billions of USD in assets. Ethereum is minimizing this disadvantage by reducing the role of the “legacy” main chain and building on a pure PoS beacon chain, which should ultimately replace the main chain. The estimated and benchmarked throughput on these initial implementations will be promising — in the thousands of TPS — but mostly meaningless in the context of a general purpose smart contract platform.

Scaling throughput while supporting non-trivial smart contracts presents challenges that neither project can claim to have solved at this point. Both are in the early stages of their roadmaps to becoming a fully sharded smart contract platform. The latest research from both projects suggests that the subsequent phase of deployments will provide some form of cross-shard communication that comes with new scaling challenges. State sharding is a solved problem, but scalable, decentralized, state sharding is not.

A key contrasting factor — that Ethereum’s blockchain forks and Zilliqa’s doesn’t — will likely result in the respective projects taking different approaches to state sharding with different tradeoffs. Based on current proposals it appears Ethereum’s approach will trade off latency for greater parallelization/scalability than Zilliqa’s, which creates a scalability bottleneck by requiring all non-trivial transactions be executed on the main chain. This may result in certain applications being better suited to one platform or the other, but longer term one can expect both platforms will continue to research innovations that enable fast, safe, and scalable support for a wide range of applications.

As a project under development Zilliqa has the advantage of taking what’s good from Ethereum’s smart contract platform and improving on the rest. The researchers at Zilliqa have applied the lessons learned in building and deploying Ethereum contracts to design a language that prioritizes safety and simplicity over expressivity. The result is an intermediate programming language that supports formal verification and can serve as a compilation target for higher level languages.

If Zilliqa can deliver a practically useful development environment (high level languages, compilers, static analysis tools, etc.) before Ethereum evolves to Vyper or some other safer language, it will present a viable and attractive alternative to Solidity. Until then the Solidity development community will be forced to accept exploits and loss of funds as a fact of life that must be tolerated because nothing better exists. When a safer alternative does exist however, projects will be financially incentivized to build on it.

The Achilles’ Heel of Ethereum might not be scalability, but the inertia of Solidity’s monopolistic success creating a general complacency toward addressing the problem that split the community once and, despite lessons learned, continues to do so.

Zilliqa’s preventive vs reactive approach to smart contract vulnerabilities may reduce the amount and complexity of governance structures required to maintain cohesion in a growing community. Even if vulnerabilities can be prevented to the extent that “code is law” becomes a viable policy, some governance mechanisms will still be needed to decide on software upgrades and parameter changes that are necessary for a platform to evolve and incorporate new innovations. Zilliqa’s governance strategy (not covered here due to lack of public information) will play an important role in attracting projects away from Ethereum and other smart contract platforms trying out various governance strategies.

Zilliqa has some major challenges to address if it is to become a high-throughput, scalable, smart contract platform. If it can address these and at the same time provide a demonstrably safer alternative to Solidity, with a functional governance process, then it may present a challenge to Ethereum’s dominance. Otherwise, it may end up being just a high-throughput simple payment network that does not compete in the smart contract space. It will be interesting to watch this project evolve through its main net launch and beyond.