Blockchain Tech — How sharding technology enables the increase of single blockchain capacity?
For the general audience, QuarkChain starts a technical column which will explain the basic knowledge of blockchain technology and decipher the fundamental technical details behind blockchains using layman language. Anyone who has technical questions or feels any “black magic’’ about blockchains can submit the concerns/comments through our bi-weekly AMA (Ask Me Anything), WeChat groups, Telegram, etc. We sincerely welcome any constructive feedback and suggestions in the journey.
The basic idea of sharding is “divide-and-conquer”, i.e., to partition a global system state into multiple sub-states and to process the transactions in each shard relatively independently. With an appropriate design of sharding technique, the capacity of the whole system is able to increase significantly.
Let’s take the assembly line as an example. A large-scale production line is able to produce the required items, whereas multiple small-scale lines can do the same with an appropriate arrangement. The rise of the production capacity is mainly attributable to the increase of the production lines. This is called sharding technology.
The Origin of Sharding Technology
It has been widely recognized that sharding was firstly proposed by Google. With the explosion of the applications of Google network, it resorted to download the entire Internet page and index it to meet the service needs of global-wise users. However, two main problems faced Google at that time: 1、Files storage; 2、How to index massive data for quick query?
In order to solve these 2 core problems, Google designed a file system GFS that is able to distribute and store a large file to hundreds of servers in the entire Internet. Afterwards, Mapreduce technique was developed to partition a large task into thousands of small tasks to distribute and process. A final result is achieved with the accumulation of all parts of results. Big Table, a system that can process and store massive amounts of structured data in real time, was developed. Thanks to these excellent systems, only a few seconds are spent to search the global webpage by using Google with much lower cost (less than 1 cent). Facebook, another world-famous company with large number of users, has also applied the same technology in order to quickly distribute User-Generated Content (UGC) .
Distributed storage and processing applied to increase the capacity of the entire system naturally adapts to meet the needs of the blockchain network. Therefore, dozens of blockchain projects resort to sharding technology to address the scalability problem. QuarkChain engineers, mostly from Google and Facebook, are well aware the value of sharding technology for blockchain projects.
The Value and Contraints of Sharding
In general, there are several advantages of sharding:
- More decentralized, which matches the main feature of blockchain]. Compared to Bitcoin’s fierce computing competition and the high performance requirements of certain projects for one node, miners are still able to get access to gain rewards by participating in mining on the shard in system applied sharding technology.
- High performance: Sharding is capable to process more transactions with parallel processing technique and allowing the increase capacity of the whole blockchain network.
- Lower cost: On one hand, the capacity of the whole network will improve with the rise of scale instead of using expensive supercomputers. In addition, gas fee will see a reduce as the increase of the capacity within the blockchain network.
- Why sharding technology fails to be widely adopted? Though papers on this technique have already published, gaps between theory and implementation still need to fill. Following this way also means that we have to build the infrastructure from scratch as there is no example in the decentralized world. With a good sharding design, the system capacity can be significantly enhanced, but achieving super high TPS (e.g. 1 million TPS) is also limited by the development of hardware technology itself. As a centralized system, Google does not need to worry about the node cheating at all. However, blockchain network, which is permissionless and mixed with good and bad participants, requires innovation on consensus algorithms and new verification mechanisms should be added to guarantee the security of the entire network.
Advantages of QuarkChain Sharding Technique
QuarkChain has proposed the decentralized sharding solution to solve the scalability problem on the basis of sharding technology applied by centralized world.
- Support state sharding: State sharding means that data on each shard will be partitioned to substates and each of them will be put onto different nodes to operate. Data of the whole network are stored on each node. The requirement of storage memory on each node is reduced.
- Guarantee security by re-shardable two-layer blockchain: QuarkChain network consists of two layers of blockchains to avoid 51% attack. We apply shards as the first layer and a root blockchain as the second layer that confirms the blocks to ensure the security and scalability.
Currently, QuarkChain testnet has successfully delivered 14k on-line TPS. The winner of TPS competition even reached 50k TPS. Currently, thousands of members have participated in QuarkChain testnet mining and the entire network is stable.
If you have any questions, please feel free to contact us! We have been holding bi-weekly AMA in telegram community. Welcome to join!