TOP Network Technical Spotlight | Two-Layer Sharding (Part 1)| Introduction
We continuously hear about the myriad of revolutionary applications that blockchain technology will bring about. Unfortunately for everyone, these applications are currently on hold. At this point, the reason for this should not be a secret. Blockchain is having a lot of trouble scaling, and this bottleneck is holding back the decentralization revolution from really taking off. It is for this reason that scalability has been at the forefront of the blockchain conversation over the past few years. While a concrete solution has yet to arise, there has been significant ground in terms of scalability research, with some very promising solutions in the pipeline.
Scaling Solutions and the Scalability Trilemma
There are two categories of scaling solutions: on-chain and off-chain. On-chain scaling solutions include new or improved consensus mechanisms, side-chains, or techniques such as sharding. While on-chain solutions are preferable, they are quickly confronted by the pesky scalability trilemma, which is extremely difficult to navigate. This predicament relates the unfortunate reality that whenever something is done to increase scalability, it is almost always accompanied with a trade-off in either decentralization, security, or both. Off-chain solutions — often called Layer-2 solutions — such as payment and state channels are not exempt from the trilemma, although they can help deal with it from another angle.
So how does TOP Network address the scalability trilemma? To reach true scalability while remaining sufficiently secure and decentralized, a single solution is likely not sufficient. TOP Network takes a more holistic approach, using several of these scaling solutions together in a complementary fashion. The three main scaling techniques utilized by TOP Network are: service-chains (similar to side-chains), a Layer-2 solution resembling state channels, and sharding. We will get to all of these topics, however the focus of this series will be on TOP’s version of sharding, and how it overcomes the many challenges sharding presents. Before we explain in detail the technology behind TOP’s implementation of sharding, let’s quickly review what sharding is in relation to blockchain systems.
What Is Sharding?
Sharding predates the advent of blockchain technology. In general, sharding is a technique used to partition a large database into many smaller, more manageable pieces called shards. Each shard is only responsible for processing a subset of the entire database, making them faster individually, and increasing the overall efficiency of the database.
How does this relate to blockchains? Well, a blockchain is really just a database — and a really inefficient one at that. A blockchain is replicated across and updated by a vast number of individual miners. Unlike a regular database, whenever an entry is to be added to a blockchain, the whole network of miners must reach a consensus on the validity of the entry — which is usually called a transaction (or block of transactions).
Since each node must process, store, and reach a consensus on the validity of every transaction that occurs in the network, the scalability of a blockchain is extremely limited. Although a blockchain network with thousands of miners may have a vast amount of combined compute resources, the maximum scalability of the blockchain is proportional to the computational power and bandwidth capabilities of just one regular mining node.
A blockchain can be thought of as a single computer where every operation must be replicated by every miner in the network. Because of this forced redundancy, no matter how many miners join the network, scalability will never increase. This is the fundamental reason why blockchains are so unscalable. On the other hand, this is also the reason why they are so secure. Since each node stores a copy of the blockchain — and all miners must reach a consensus on the validity of every transaction before appending it — tampering with a blockchain ledger is exceedingly hard.
Sharding presents a way of increasing the throughput of a blockchain by divvying up the work between participating miners, therefore reducing redundancy. Instead of requiring every node to validate every transaction, miners can be partitioned into smaller groups that process transactions separately. This allows transactions to be validated in parallel instead of serially (one after another).
Since blockchains are quite a bit more complex than regular databases, there actually turns out to be several different levels of sharding as it relates to blockchain systems.
- State Sharding
The state of a blockchain is specified by a set of parameters which could vary depending on the blockchain. Commonly, the full state is given by the latest account balances of all addresses, the most recent state of every smart-contract deployed on the blockchain, and a few other things. In short, the state of a blockchain is the latest valid snapshot of essentially every important property related to that blockchain network. Without state sharding, every mining node in the network must store the complete state.
State sharding is a form of sharding wherein smaller groups of miners store a subset of the full state. Analogous to database sharding, state sharding involves splitting up the entire blockchain into many smaller pieces, and subsequently assigning each of these pieces to be stored by a designated group of miners.
The benefit is that miners do not need to store the entire blockchain, which can grow to a prohibitively large size over time. State sharding could also be called storage sharding.
- Computation Sharding
In a non-sharded blockchain network, every miner must reach a consensus regarding the validity of each and every transaction with the entire network of miners.
With compute sharding, the consensus network of miners is first divided into groups — typically given the name “shards.”
Each group of miners in each shard is only responsible for validating and reaching consensus on a certain subset of transactions. Determining which transactions each shard is responsible for depends upon the implementation. With an account/balance model, each shard is responsible for a certain range of accounts (address subspace), meaning nodes only care about transactions sent to or from addresses designated to their shard.
This allows transactions to be processed in parallel, which greatly increases throughput while distributing the compute load. However, without state sharding, each node would still have to store the entire blockchain, which would grow even faster with the increased TPS from parallelization.
- Network Sharding
State sharding spreads the storage load between shards so that no miner has to store the full blockchain. Compute sharding divides among shards the computational load required from transaction validation, smart-contract execution, and consensus. However, there is still one remaining bottleneck — bandwidth.
The bandwidth load on each node increases rapidly as the number of shards in the network grows. Increasing the number of shards is desirable because it increases throughput, though increasing TPS inadvertently leads to escalating bandwidth requirements for nodes. Furthermore, adding shards results in a higher proportion of transactions being cross-shard, meaning they require communication between two different shards. If the load from routing transactions/blocks and keeping account states synchronized across all shards is not distributed efficiently, the bandwidth requirements for nodes would become unreasonably high.
Network Sharding divides the routing and synchronization duties between many groups of nodes — which are usually of a non-mining variety. Without Network Sharding, no single node would have the bandwidth necessary to send/receive enough packets in a blockchain network executing hundreds of thousands of TPS. Furthermore, requiring the consensus/mining nodes to route transactions and synchronize states in conjunction to their already high computational load could lead to centralization, as running a node would become too costly.
One way to mitigate this is to include nodes in the network who do not participate in consensus, but instead are dedicated to routing cross-shard transactions and keeping account states synchronized across shards. Dividing the bandwidth load between many groups of this new type of routing node ensures that no single node is prohibitively expensive to run, which helps in keeping the network permissionless. We will learn more about this when discussing the details of TOP’s two-layer sharding technology in the upcoming parts.
TOP Network’s Approach
There are quite a few blockchain projects currently working on sharding. A large number of these projects only plan to employ compute sharding (sometimes called transaction sharding by i.e Zilliqa), which is generally much easier than state sharding. Adding smart-contract functionality makes state sharding extremely difficult, which is why many projects are veering away from it. Ethereum 2.0 will attempt to apply state sharding — which includes transaction/compute sharding — although there are many intermediate upgrades to Ethereum 1.0 necessary before this becomes closer to reality.
TOP Network will implement state sharding, compute sharding, and network sharding. Using a unique two-layer sharding technique with a block-lattice data structure, TOP can confront many of the issues that make state sharding so difficult. The team behind TOP Network leverages their 20+ years of experience developing communications networks and other distributed systems to approach sharding from a new perspective, helping to solve some of the major issues surrounding cross-shard transaction routing, cross-shard synchronization, and bandwidth bottlenecks.
Stay tuned for part 2 where we will dig deeper into TOP Network’s comprehensive implementation of sharding.