Overlord — A new consensus algorithm
Overlord is a consensus algorithm customized for Huobi Chain, that was researched and developed by a Nervos development team.
Huobi Chain, a public, regulatory-friendly blockchain focused on decentralized financial (DeFi) services— utilizes Overlord—and launched in public beta earlier this week
Yun Zhou, Project Manager at Nervos working on Huobi Chain, shared the design process and features integral to Overlord:
Before selecting a consensus algorithm for the Huobi public chain, we first asked ourselves:
- Who is the Huobi Chain serving?
- What problem does it solve?
- What social value is being created?
To this end, we requested input from professionals in the fields of finance and management, followed the real needs of the market, and listened carefully to the feedback from the community. In June 2019, we made significant progress on all fronts, and shared our vision in the article “Nervos and Huobi Partner For The Future Decentralized Economy”.
Today, financial markets and blockchain technology applications are relatively separated. Huobi Chain is positioned as an efficient and global financial market infrastructure based on the blockchain. “Efficient” corresponds to high-speed bursts of on-chain transactions; “Global” corresponds to the friendly compatibility and high scalability of the underlying chain; “Infrastructure” means that the development of on-chain applications is standardized and streamlined, which facilitates a rich developer ecosystem in the future.
A blockchain’s consensus functions like a heart; it must be both efficient and powerful to allow applications built on the chain to flourish: Huobi public chain requires a Byzantine fault tolerant (BFT) consensus algorithm that can support thousands of transactions per second with hundreds of consensus nodes, and transaction delays of no more than a few seconds.
Through our research, we investigated the common modern BFT consensus algorithms: PBFT, Tendermint and Hotstuff. Unfortunately, none of these consensus algorithms met our needs.
- PBFT, the earliest practical Byzantine fault-tolerant consensus algorithm has a problem resulting from excessive message complexity — O(n³) which is caused by how it handles view changes (the process of switching leaders).
- Tendermint is a consensus algorithm proposed in 2017. At its core, it adopts a “lock-unlock” mechanism, which simplifies the leader-switching process. However, this decision results in a loss of responsiveness, which means that even under excellent network conditions, Tendermint consensus still requires a fixed block time. Though view changes are simplified and message complexity is reduced to O(n²), it still did not meet our requirement of higher consensus efficiency.
- Hotstuff came into the public eye via Libra in 2019. It makes many improvements in message complexity and responsiveness, however transaction validation requires at least three blocks. This limitation is significant and cannot meet the needs of specific businesses scenarios under certain circumstances.
So, we decided to build a new consensus algorithm—Overlord— to meet our specific needs.
In testing, we found that Overlord can support hundreds of consensus nodes and thousands of transactions per second. In addition, the transaction delay is no more than a few seconds and transactions cannot be rolled back on the chain.
The design and development phases of the Overlord algorithm have been completed, and the initial engineering implementation has been open sourced on Github.
How does Overlord achieve these properties？
First, Overlord introduces a mechanism that utilizes BLS aggregate signatures to collect consensus votes from nodes. A node no longer broadcasts consensus messages to other nodes through the P2P network. Instead, it sends the message to the leader, which will process it and send to other nodes. Through this mechanism, the communication complexity of the system has been greatly reduced-—from O(n²) to O(n)— significantly improving consensus efficiency.
Additionally, Overlord adopts a “timeout + brake” mechanism to address the issue of malicious leaders. This significantly speeds up the consensus process and reduces the difficulty of implementation. With this design, the number of messages required for consensus is greatly reduced, decreasing the proportion of bandwidth used by these messages (for voting, etc.), further improving consensus efficiency.
In Overlord, block verification and voting processes happens in parallel. Compared with other consensus algorithms, Overlord has higher block generation speed and greater transaction processing capacity.
The most important property of Overlord is its decoupling of transaction ordering and state consensus, which allows execution and consensus processes to run in parallel. This design greatly reduces idle time, improving consensus efficiency compared with Tendermint (which requires waiting for the execution result before proceeding to the next block). In Overlord, once a block has been accepted into the chain, it can be considered confirmed and any transactions in that block cannot be rolled back. This fast finality is necessary for financial or business use cases with a large number of transactions or complex contracts.
Why the name “Overlord?”
Developers and Gamers have a lot in common, and Nervos Devs are no different. In Starcraft, an Overlord is a ‘significantly evolved creature’ that psychically controls swarms of the Zerg population, similar to the nature of consensus we understand. If a group of nodes is imagined as a swarm, Overlord is the brain of the swarm. In Starcraft, an Overlord functions similarly to the consensus process of a blockchain, which controls the functions of all nodes.
We believe that through the utilization of Overlord consensus, we can build a highly performant and widely distributed blockchain system, and Overlord will become a common consensus solution for the next generation of high performance blockchains.
For more about Overlord, please check GitHub.
