Lightning Fast Consensus Becomes a Reality Thanks to Fetch.ai Breakthrough
The finality problem is a technical limitation of blockchains that has acted as a barrier to wider adoption of the technology. In layman’s terms, the finality time can be thought of as the wait between a transaction first being submitted to the network and it being confirmed as being valid. To succeed in revolutionizing our economies, blockchains must achieve time-to-finality that provides a comparable experience to centralized payment systems.
At Fetch.ai, we have been working on the finality problem, and are delighted to announce a cryptographic breakthrough that will enable dramatic improvements to the speed at which transactions are confirmed, while also increasing security. Our recently published paper explains how this is achieved. In this Medium article, we outline the history that has led to the state-of-the-art in consensus design and how our cryptographic innovation moves the field forward.
Anyone who has held Bitcoin will be familiar with the one-hour wait for their transactions to be confirmed. Compared to gold, or a physical asset that serves as a stable store-of-value, Bitcoin transactions are relatively fast and low-cost. At the same time, no-one would be willing to wait for an hour to buy a cup of coffee. And the economic systems of the future, such as networks of IoT devices, will require micro-transactions that are several orders of magnitude faster and cheaper than is possible with any first-generation blockchain.
More recent blockchain designs have sought to overcome the expense and inefficiency of Bitcoin’s Proof-of-Work (PoW) consensus protocol by switching to Proof-of-Stake (PoS), where nodes gain entry to consensus by staking tokens rather than proving that energy (or work) has been expended. One advantage of PoS is that it enables blocks to be produced at a rate that is more regular and therefore faster than is possible with PoW.
Rapid block production means that transactions can be entered into a block more quickly, which is one way of speeding up their confirmation. The other way that PoS speeds up transaction finality is by introducing determinism so that a block becomes finalized after a smaller number of blocks have been added to the chain. Bitcoin’s probabilistic definition of finality means that a transactions can never be guaranteed as confirmed, but six blocks has become the operational definition of finality that is accepted by most users of Bitcoin.
The hour-long wait for Bitcoin transaction finality arises from the requirement for six block confirmations along with the block time of approximately ten minutes.
finality time = block confirmations × block time
In most PoS systems, the number of block confirmations is at most two, while the block production time can be reduced to seconds thereby leading to finality times that are also on the order of seconds. Performance at this speed would allow blockchains to compete with centralized payment systems. These centralized systems use a settlement process that takes days to complete but create an illusion of speed through the use of trusted communication channels between the merchants, payment providers and banks. It’s important to note that most aspects of blockchain design involve different trade-offs. Having outlined the advantages of PoS, we now go on to discuss some of its limitations and how Fetch.ai’s novel cryptographic scheme pushes forward the capabilities of the technology.
The feature of PoS schemes that enables deterministic finality is the public record of the validators that are eligible to participate in consensus. Since the identity of validators is known, it’s possible for them to communicate with each other to establish the validity of a block before it is added to the chain. This idea is the basis of a family of protocols based on practical Byzantine Fault Tolerance (pBFT) that underpin cryptocurrencies such as EOS, Neo and Tendermint.
These pBFT-derived protocols provide instant finality but introduce two additional problems. The first of these is that the additional communication overhead restricts the number of validators that the consensus can support. This depends on the details of the consensus but the upper limit ranges from a handful to a maximum of perhaps a few hundred validators. The second problem is that the order in which the validators propose blocks typically occurs with a regular pattern, so the identity of the next block producer is known in advance. This leaves the validator vulnerable to Distributed Denial-of-Service (DDoS) attacks that prevent it from communicating with the rest of the network. These DDoS attacks can be resisted by isolating the consensus node behind a load-balanced wall of sentry nodes that enable it continue operating during the attack. However, this substantially increases the cost of running a validator node.
An alternative to pBFT is to allow validators to propose blocks and then use voting (or notarization) to reach agreement on which blocks are valid. This approach can be applied to blockchains or a transient Directed Acyclic Graph (DAG) that is used in the Fetch.ai minimal agency consensus. In this family of protocols, the predictable pattern of block production is replaced by an unbiased, cryptographically secure Decentralized Random Beacon (DRB) that hides the identity of the next block producer until the very last moment. This replicates the randomness inherent to PoW mining but without the enormous financial and environmental cost. This feature, along with built-in redundancy over which of validators are authorized to produce a block, makes these protocols substantially more resistant to DDoS attacks, allows them to support a larger number of validators and typically increases block production rates compared with BFT protocols.
The core of the DRB is a cryptographic protocol that involves aggregating messages from the population of validators into a single pseudo-random value. Pseudo-randomness means that the value is deterministic but has similar properties to a truly random process. The other key property of the DRB is that it cannot be predicted in advance by any individual validator. This process appears almost “magical”, but these techniques have a very sound mathematical basis, which we will be explaining in more detail in a later article.
The Fetch.ai DRB advances over the previous state-of-the-art BLS scheme in two ways. The first is that it enables the DRB to be calculated a factor of at least three times faster, thereby increasing the maximal rate at which blocks can be created. This improves both throughput and transaction finality. The second advantage is that our DRB’s security has been proven using a more rigorous cryptographic model, which limits the scope for an attacker to compromise the system. We’re currently preparing the paper, describing the cryptographic scheme, for peer-review at a major security conference. We anticipate that this innovation, along with application of multi-agent system theory to optimize costs, will enable the Fetch.ai blockchain to achieve its goal of enabling the future digital economy.
At the same time, we very much believe that a rising tide lifts all boats and that delivering the benefits of blockchain technology is far more important than the success of any individual project. For this reason, we have provided an optimized and thoroughly documented C++ implementation of our DRB on Github that we are releasing under an Apache 2.0 license. In the coming weeks, we are also planning on conducting an external security audit and will be providing bindings to other programming languages that allow the technology to be applied as widely as possible. Finally, we will also be reaching out to other projects that we admire to share the benefits of this new and exciting technology.
If you are interested in integrating our DRB into your project, I would be happy to hear your thoughts and comments on Telegram or Twitter. As I mentioned earlier, I’ll also be writing another Medium post in the next few weeks describing more of the theory behind the DRB so stay tuned for further updates.
