Demystifying Blockchain and Consensus Mechanisms — Everything You Wanted to Know But Were Never Told

It is likely that you’ve heard so far, many descriptions of what blockchain is, and that description probably is related somehow with money. Of course, this is not happening by chance, but actually due to many popular technologies such as Bitcoin, Ethereum, Ripple and many others currently available in the cryptocurrency marketplace, which have this solution based on DLT(Distributed Ledger Technology), as their core implementation foundation, which is the basis for trading cryptocurrencies and other assets through public & private markets. However, Blockchain technology goes much further than just cryptocurrencies. Today, blockchain is already adopted as part of many everyday B2B transactions, including those powered by enterprise applications such as ERPs, Supply Chain, Financial Services, Healthcare systems, etc, and the list is much longer than this one.

The Blockchain is an undeniably ingenious invention — the brainchild of a person or group of people known by the pseudonym, Satoshi Nakamoto. But since then, it has evolved into something greater, and the main question every single person is asking is: What is Blockchain?

By definition, Blockchain is a continuously growing list of records, called blocks, which are linked and secured using cryptography. Each block typically contains a cryptographic hash of the previous block, a timestamp and transaction data. By design, a blockchain is inherently resistant to modification of the data. It is “an open, distributed ledger that can record transactions between two parties efficiently and in a verifiable and permanent way”. For use as a distributed ledger, a blockchain is typically managed by a peer-to-peer network collectively adhering to a protocol for validating new blocks. Once recorded, the data in any given block cannot be altered retroactively without the alteration of all subsequent blocks, which requires collusion of the network majority.

Generally speaking, a blockchain network is a system for maintaining distributed ledgers or facts and the history of the ledgers’ updates. This approach allows organizations that don’t fully trust each other to agree on the updates submitted to a shared ledger by using peer-to-peer protocols rather than a central third party or manual offline reconciliation process. Blockchain enables real-time transactions and securely shares tamper-proof data across a trusted business network.

There are essentially two types of Blockchain considering the access perspective:

Permissionless

Basically, anyone can read the chain, anyone can make legitimate changes and anyone can write a new block into the chain (as long as they follow the rules).

Bitcoin is by far the most popular example of a successful public blockchain network. It is totally decentralized. It is also described as a ‘censor-proof’ blockchain. Bitcoin and other cryptocurrencies such as Ethereum currently secure their blockchain by requiring new entries to include a proof of work. However, due to the way a public blockchain works, they require computer resource intensive mining process to add blocks cryptographically. Also, Consensus models based on computationally expensive algorithms requiring the processing power of many nodes to ensure security

The great advantage to an open, permissionless, or public, blockchain network is that guarding against bad actors is not required and no access control is needed. This means that applications can be added to the network without the approval or trust of others, using the blockchain as a transport layer.

For these reasons, it’s also known by its widest description, a public blockchain. But, obviously, this is not the only way to build a blockchain.

Permissioned

Essentially they are a closed ecosystem where members are invited to join and keep a copy of the ledger. e.g.: (Hyperledger, R3 Corda). Permissioned blockchains use an access control layer to govern who has access to the network. In contrast to public blockchain networks, validators on private blockchain networks are vetted by the network owner.

They do not rely on anonymous nodes to validate transactions nor do they benefit from the network effect, but they rely on something called consensus protocol, like bitcoin’s proof of work (the one we hear about most often), that does two basic things: it ensures that the next block in a blockchain is the one and only version of the truth, and it keeps powerful adversaries from derailing the system and successfully forking the chain.

Consensus protocol comprises of 3 basic steps:

• Endorsement: determine whether to accept or reject a transaction
• Ordering: sort all transactions within a time period into a sequence
• Validation: verify endorsement satisfy policy and Read set is valid

Permissioned networks can also go by the name of ‘consortium’ or ‘hybrid’ blockchains.

Blockchain Consensus mechanisms

A blockchain is a decentralized peer-to-peer system with no central authority figure. While this creates a system that is devoid of corruption from a single source, it still creates a major problem.

• How are any decisions made?
• How does anything get done?

Think of a normal centralized organization. All the decisions are taken by the leader or a board of decision makers. This isn’t possible in a blockchain because a blockchain has no “leader”. For the blockchain to make decisions, they need to come to a consensus using “consensus mechanisms”.

So, how do these consensus mechanisms work and why did we need them? What are some of the consensus mechanisms used in cryptocurrencies and in some Blockchain implementations such as Hyperledger? All these questions will be answered later on, however let’s understand how a consensus work prior to talk about some available implementations.

In simpler terms, consensus is a dynamic way of reaching agreement in a group. While voting just settles for a majority rule without any thought for the feelings and well-being of the minority, a consensus on the other hand makes sure that an agreement is reached which could benefit the entire group as a whole. A method by which consensus decision-making is achieved is called “consensus mechanism”.

So now that we have defined what a consensus is, let’s look at what the objectives of a consensus mechanism are:

• Agreement Seeking: A consensus mechanism should bring about as much agreement from the group as possible.
• Collaborative: All the participants should aim to work together to achieve a result that puts the best interest of the group first.
• Cooperative: All the participants shouldn’t put their own interests first and work as a team more than individuals.
• Egalitarian: A group trying to achieve consensus should be as egalitarian as possible. What this basically means that each and every vote has equal weight. One person’s vote can’t be more important than another’s.
• Inclusive: As many people as possible should be involved in the consensus process. It shouldn’t be like normal voting where people don’t really feel like voting because they believe that their vote won’t have any weight in the long run.
• Participatory: The consensus mechanism should be such that everyone should actively participate in the overall process.

Now that we have defined what consensus mechanisms are and what they should aim for, we need to think of the other questions: Which consensus mechanisms should be used for blockchain network to keep their original characteristics such reliability, security and availability?

We hear plenty of talk of how public blockchains are going to change the world, but to function on a global scale, a shared public ledger like Bitcoin needs a functional, efficient and secure consensus algorithm.

Before Bitcoin, there were loads of iterations of peer-to-peer decentralized currency systems which failed because they were unable to answer the biggest problem when it came to reaching a consensus. This problem is called “Byzantine Generals Problem(BGP)”.

Byzantine Generals Problem(BGP)

Imagine that several divisions of the Byzantine army are camped outside an enemy city, each division commanded by its own general. The generals can communicate with one another only by messenger. After observing the enemy, they must decide upon a common plan of action. However, some of the generals may be traitors, trying to prevent the loyal generals from reaching agreement. The generals must decide on when to attack the city, but they need a strong majority of their army to attack at the same time. The generals must have an algorithm to guarantee that: (a)

1. All loyal generals decide upon the same plan of action, and
2. A small number of traitors cannot cause the loyal generals to adopt a bad plan. The loyal generals will all do what the algorithm says they should, but the traitors may do anything they wish.

The algorithm must guarantee condition (a) regardless of what the traitors do. The loyal generals should not only reach agreement, but should agree upon a reasonable plan.

Figure 3 — Byzantine Generals Problem

Looking at the picture above, you can understand the problem and what is the challenge for Byzantine generals while attacking a city. They are facing two very distinct problems:

• The generals and their armies are very far apart so centralized authority is impossible, which makes coordinated attack very tough.
• The city has a huge army and the only way that they can win is if they all attack at once.

What these generals need, is a consensus mechanism which can make sure that their army can actually attack as a unit despite all these setbacks.

This has clear references to blockchain as well. The chain is a huge network; how can you possibly trust them? If you were sending someone 4 Bitcoin from your wallet, how would you know for sure that someone in the network isn’t going to tamper with it and change 4 to 40 Bitcoins?

This is where consensus mechanisms come to the rescue. As such, now we are going to go through a list of consensus mechanisms which can solve the Byzantine Generals problem for some very known Blockchain networks such as Bitcoin, Ethereum, Ripple, Peercoin, Hyperledger and many others.

Proof of Work (PoW)

Bitcoin uses Proof of Work(PoW) to ensure blockchain security and consensus. “Proof of Work”, as its name implies, requires that the decentralized participants that validate blocks show that they have invested significant computing power in doing so.

In bitcoin, validators (known as “miners”) compete to process a block of transactions and add it to the blockchain.

In proof of work, miners compete to add the next block (a set of transactions) in the chain by racing to solve a extremely difficult cryptographic puzzle. They do this by churning enough random guesses on their computer to come up with an answer within the parameters established by the bitcoin. This process requires immense amount of energy and computational usage. The puzzles have been designed in a way which makes it hard and taxing on the system. Essentially this puzzle that needs solving is to find a number that, when combined with the data in the block and passed through a hash function, produces a result that is within a certain range. This is much harder than it sounds. The main character in this game is called a “nonce”, which is an abbreviation of “number used once”. In the case of bitcoin, the nonce is an integer between 0 and 4.294.967.296.

How do they find this number? By guessing at random. The hash function makes it impossible to predict what the output will be. So, miners guess the mystery number and apply the hash function to the combination of that guessed number and the data in the block. The resulting hash has to start with a pre-established number of zeroes. There’s no way of knowing which number will work, because two consecutive integers will give wildly varying results. What’s more, there may be several nonces that produce the desired result, or there may be none (in which case the miners keep trying, but with a different block configuration).

When a miner solves the puzzle, they present their block to the network for verification. Verifying whether the block belongs to the chain or not is an extremely simple process. The first to solve the puzzle, wins the lottery. As a reward for his or her efforts, the miner receives newly bitcoins — and a small transaction fee.

The difficulty of the calculation (the required number of zeroes at the beginning of the hash string) is adjusted frequently, so that it takes on average about 10 minutes to process a block.

Why 10 minutes? That is the amount of time that the bitcoin developers think is necessary for a steady and diminishing flow of new coins until the maximum number of 21 million is reached (expected some time in 2140).

Yet, although a masterpiece in its own right, bitcoin’s proof of work isn’t quite perfect.

Common criticisms include that it requires enormous amounts of computational energy, that it does not scale well (transaction confirmation takes about 10–60 minutes) and that the majority of mining is centralized in areas of the world where electricity is cheap, leading to an inefficient process because of the sheer amount of power and energy that it eats up.

That said, people and organizations that can afford faster and more powerful ASICs(Application-specific integrated circuit chips) usually have better chance of mining than the others. As a result of this, bitcoin isn’t as decentralized as it wants to be. Theoretically speaking, there are big mining pools that could simply team up with each other and launch over than 51% on the bitcoin network. As a result, those who have significant financial resources have come to dominate the bitcoin mining space. Mining today is embodied by the emergence of enterprise-style, datacenter-hosted mining operations.

Bitcoin creator Satoshi Nakamoto woke us up to the potential of the blockchain, but that doesn’t mean we can’t keep searching for faster, less centralized and more energy-efficient consensus algorithms to carry us into the future. Other examples can be find below such as PoS(Proof-of-Stake), Proof-of-Activity and some others available today.

Proof-of-Stake (PoS)

The most common alternative to proof of work is proof of stake. In this type of consensus algorithm, instead of investing in expensive computer equipment in a race to mine blocks, a ‘validator’ invests in the coins of the system. Note the term validator. That’s because no coin creation (mining) exists in proof of stake. Instead, all the coins exist from day one, and validators (also called stakeholders, because they hold a stake in the system) are paid strictly in transaction fees. The systems that don’t use proof-of-work are also often called virtual mining systems because they don’t have a mining activity.

The network selects an individual to approve new messages (that is to say, confirm the validity of new information submitted to the databse) based on their proportional stake in the network. In other words, instead of any individual attempting to calculate a value in order to be chosen to establish a consensus point, the network itself runs a lottery to decide who will announce the results, and system participants are exclusively and automatically entered into that lottery in direct proportion to their total stake in the network. As in the PoW system run by Bitcoin, the PoS system run by organizations such as Peercoin also provides an incentive to participation, which ensures broadest possible network participation and therefore the most robust network security possible. In the Peercoin system, the chosen party is rewarded with a new Peercoin in a process called ‘minting’ (rather than BitCoin’s ‘mining’).

As mentioned, proof of stake will make the entire mining process virtual and replace miners with validators. Here is an outline on how the process will work:

• The validators will have to lock up some of their coins as stake.
• After that, they will start validating the blocks. Meaning, when they discover a block which they think can be added to the chain, they will validate it by placing a bet on it.
• If the block gets appended, then the validators will get a reward proportionate to their bets.

In proof of stake, your chance of being picked to create the next block depends on the fraction of coins in the system you own (or set aside for staking). A validator with 300 coins will be three times as likely to be chosen as someone with 100 coins.

Once a validator creates a block, that block still needs to be committed to the blockchain. Different proof-of-stake systems vary in how they handle this. There are some implementations where every node in the system has to sign off on a block until a majority vote is reached, while in other systems, a random group of signers is chosen.

As you can see, the PoS protocol is a lot more resource-friendly than PoW. In PoW, you NEED to waste a lot of resources to go along with the protocol, it is basically resource wastage for the sake of resource wastage.

Although PoS seems to be the most reasonable replacement for PoW, due to not having the issues found in PoW(requires enormous amounts of computational energy, not decentralized as it wants to be since there are just a few large pools that own over than 50% of Bitcoin network together), there is a very common problem that needs to be solved by PoS prior to be largely adopted by a production blockchain implementation. So, reviewing the way PoS works with regards to security, the common questions that could arise would be the following: What is to discourage a validator from creating two blocks and claiming two sets of transaction fees? And what is to discourage a signer from signing both of those blocks? This has been called the ‘nothing-at-stake’ problem. A participant with nothing to lose has no reason not to behave badly. In the burgeoning field of ‘crypto-economics’, blockchain engineers are exploring ways to tackle this and other problems. One answer is to require a validator to lock their currency in a type of virtual vault. If the validator tries to double sign or fork the system, those coins are slashed.

Additionally, this system, however, by rewarding those who already are most deeply involved in the network inherently creates an increasingly centralized system. This is inimical to a truly robust network. Therefore proponents of PoS systems have put forward a number of various modifications to help ensure the base for their networks remain as broad (and therefore secure) as possible.

Peercoin was the first coin to implement proof of stake. Ethereum currently relies on proof of work, but is planning a move to proof of stake in early 2018 by solving the PoS problem called ‘nothing-at-stake’ by leveraging a new approach to address this PoS issue called Casper protocol.

Also, there is a variation of this method called a delegated proof-of-stake (DPoS). This system works along the same lines as the PoS system, except that individuals choose an overarching entity to represent their portion of stake in the system. So imagine, each individual decides if entity 1, 2, or 3 (these could be, for example, computer servers, and are called ‘delegate nodes’ within a DPoS system) will ‘represent’ his or her individual stake in the system. This allows individuals with smaller stakes to team up to magnify their representation, thereby creating a mechanism to help balance out the power of large stake holders. This comes at the cost, however of greater network centralization. Bitshares is one company that employs a DPoS system.

Proof-of-Activity(PoA)

So, proof of activity was created as an alternative incentive structure for bitcoin. Proof of activity is a hybrid approach that combines both proof of work and proof of stake. In proof of activity, mining kicks off in a traditional proof-of-work fashion, with miners racing to solve a cryptographic puzzle. Depending on the implementation, blocks mined do not contain any transactions (they are more like templates), so the winning block will only contain a header and the miner’s reward address.

At this point, the system switches to proof of stake. Based on information in the header, a random group of validators is chosen to sign the new block. The more coins in the system a validator owns, the more likely he or she is to be chosen. The template becomes a full-fledged block as soon as all of the validators sign it. If some of the selected validators are not available to complete the block, then the next winning block is selected, a new group of validators is chosen, and so on, until a block receives the correct amount of signatures. Fees are split between the miner and the validators who signed off on the block.

Criticisms of proof of activity are the same as for both proof of work (too much energy is required to mine blocks) and proof of stake (there is nothing to deter a validator from double signing). Decred is the only coin right now using a variation of proof of activity.

Practical Byzantine Fault Tolerance Algorithm(PBFT)

The Practical Byzantine Fault Tolerance Algorithm (PBFT) was designed as a solution to a problem presented in the form of an allegory described earlier in this introduction chapter under the Byzantine Generals Problem(BGP) section.

To clarify the allegory for our purposes: the ‘generals’ in the story are the parties participating in the distributed network running the blockchain (database) in question. The messengers they are sending back and forth are the means of communication across the network on which the blockchain is running. The collective goal of the “loyal generals” is to decide whether or not to accept a piece of information submitted to the blockchain (database) as valid or not. A valid piece of information would be, in our allegory, a correct opportunity to decide in favor of attack. Loyal generals, for their part, are faithful blockchain participants, who are interested in ensuring the integrity of the blockchain (database) and therefore ensuring that only correct information is accepted. The trecherous generals, on the other hand, would be any party seeking to falsify information on the blockchain (the database). Their potential motives are myriad — it could be an individual seeking to spend a BitCoin that she does not actually own or another person who wants to get out of contractual obligations as outlined in a smart contract he already signed and submitted.

Various computer scientists have outline a number of potential solutions to the Byzantine generals problem from the allegory. The practical byzantine fault tolerance algorithm (PBFT), which is used to establish consensus in blockchain systems, is only one of those potential solutions. Three examples of blockchains that rely on the PBFT for conses are Hyperledger, Stellar, and Ripple. Very roughly and without explaining the whole algorithm (which would take a multiple page research paper), what the PBFT does is as follows: Each ‘general’ maintains an internal state (ongoing specific information or status). When a ‘general’ receives a message, they use the message in conjunction with their internal state to run a computation or operation. This computation in turn tells that individual ‘general’ what to think about the message in question. Then, after reaching his individual decision about the new message, that ‘general’ shares that decision with all the other ‘generals’ in the system. A consensus decision is determined based on the total decisions submitted by all generals.

Among other considerations, this method of establishing consensus requires less effort than other previous methods described earlier. Also, PBFT is a system initially devised for low-latency storage systems — something that could be applicable in digital asset-based platforms that don’t require a large amount of throughput, but do demand many transactions.

Hyperledger’s approach for consensus

The Hyperledger project allows developers to create their own digital assets with a distributed ledger powered by nodes built on the principle of PBFT. The system could be used to digitally back a real asset (such as a house), create new coins, or form a fault-tolerant system of consensus.

The idea for Hyperledger’s use of PBFT goes beyond asset-based systems. It takes the idea of an algorithm for consensus and uses it to distribute all sorts of technical solutions — not just the low latency, high-speed file storage solution it was originally built to provide. This might be a good method of testing the power of nodes that do not use incentive to develop their strength. What will happen without such rewards? Systems like Hyperledger aim to find out. If you use Byzantine Fault Tolerance, ideally corruption problems are contained. The other nodes can realize a node is misbehaving, and not respond to its messages.

In distributed ledger technology, consensus has recently become synonymous with a specific algorithm, within a single function. However, consensus encompasses more than simply agreeing upon the order of transactions, and this differentiation is highlighted in Hyperledger Fabric through its fundamental role in the entire transaction flow, from proposal and endorsement, to ordering, validation and commitment. In a nutshell, consensus is defined as the full-circle verification of the correctness of a set of transactions comprising a block.

As for Hyperledger implementation, consensus is ultimately achieved when the order and results of a block’s transactions have met the explicit policy criteria checks. These checks and balances take place during the lifecycle of a transaction, and include the usage of endorsement policies to dictate which specific members must endorse a certain transaction class, as well as system chaincodes to ensure that these policies are enforced and upheld. Prior to commitment, the peers will employ these system chaincodes to make sure that enough endorsements are present, and that they were derived from the appropriate entities. Moreover, a versioning check will take place during which the current state of the ledger is agreed or consented upon, before any blocks containing transactions are appended to the ledger. This final check provides protection against double spend operations and other threats that might compromise data integrity, and allows for functions to be executed against non-static variables.

Also, since Hyperledger Fabric requires all participants to be authenticated, due to its permissioned implementation nature, in addition to take advantage of this characteristic to govern certain levels of access control (e.g. this user can read the ledger, but cannot exchange or transfer assets), it also can benefit on this dependence on identity as a great advantage in that varying consensus algorithms (e.g. byzantine or crash fault tolerant) can be implemented in place of the more compute-intensive Proof-of-Work and Proof-of-Stake varieties, as it was properly described earlier in this section. As a result, permissioned networks tend to provide higher transaction throughput rates and performance.

In addition to the multitude of endorsement, validity and versioning checks that take place, there are also ongoing identity verifications happening in all directions of the transaction flow. Access control lists are implemented on hierarchical layers of the network (ordering service down to channels), and payloads are repeatedly signed, verified and authenticated as a transaction proposal passes through the different architectural components. To summarize, consensus is not merely limited to the agreed upon order of a batch of transactions, but rather, it is an overarching characterization that is achieved as a byproduct of the ongoing verifications that take place during a transaction’s journey from proposal to commitment.

Conclusion about consensus mechanisms

While these systems for establishing consensus are currently the most dominant, the field is still wide open to innovation by creating variations of these implementations as well as new approaches to them. Some other examples are: Proof of burn, Proof of capacity, and Proof of elapsed time. As blockchain systems continue to gain in popularity, they will also continue to grow in scale and complexity. Which of these consensus building systems (if any) is best equipped to handle this ongoing expansion remains to be seen. Currently, companies choose a system for their product that best meets their (or their customer’s) needs for speed, efficiency, and security.

It is important to note, these systems differ not only in the details of the formation of their respective consensus-building communities, but importantly they differ in how they would handle potential attacks. This is, in fact, one of the clearest distinguishing features between the consensus-building systems: the potential size of an attack on the system that could be easily managed.

If you’ve made it this far, then congratulations! There is still so much more to explain about the Blockchain and Hyperledger, but at least now you have an idea of the broad outline of the genius of the programming and the concept. For the first time we have a system that allows for convenient digital transfers in a decentralized, trust-free and tamper-proof way. Sky is the limit for Blockchain!!

Please clap on this article if it has helped you to understand a bit about blockchain and consensus mechanisms. And please leave a comment and state your feedback. This is very important to help me keep writing new articles about this sort of content.