Forks, Signaling, and Activation
There seems to be a considerable amount of confusion among Bitcoin users regarding the nature of the process underlying changes to the Bitcoin protocol. In an earlier post I described the process for making code changes to the Bitcoin Core software repository. In this post I’ll go over the far more intricate process underlying changes to the consensus rules that ensure everyone ends up on the same blockchain and sees the same ledger and transaction history.
Different nodes ending up on different chains is known as a consensus fork or a chain split. Note that this type of fork is very different than a codebase fork (usually done by cloning someone else’s repository). Most codebase forks (if done correctly) will not result in nodes ending up on different chains unless that is the desired intent, as in the case of altcoins.
When different nodes end up on different chains either unintentionally or as a result of an attack, it is called a consensus failure. The property of antifragility rests strongly on the network’s resilience to consensus failures.
By far the vast majority of changes to the Bitcoin codebase have no impact on consensus rules.
Some changes involve just moving code around or renaming things to clean things up, also known as refactoring. A good example of this type of change is Matt Corallo’s removal of the main.h and main.cpp files.
Other changes involve optimization of certain operations to either speed them up or to lower other resource requirements, such as space or memory. Examples of these kinds of changes include Pieter Wuille’s introduction of headers-first synchronization which dramatically reduced initial synchronization time when starting up a new node as well as the use of the secp256k1 crypto library (also mostly authored by Wuille) which greatly sped up ECDSA signature validation making it take significantly less time to validate transactions and blocks and is now probably better tested and more secure than the OpenSSL library which it replaced.
The examples mentioned thus far had significant beneficial impact on the software but generally had no impact on compatibility with other nodes.
When a change is proposed that affects compatibility with other nodes, a BIP (Bitcoin Improvement Proposal) is typically required. A brief description of the BIP process along with a list of existing BIPs can be found here.
In order to help classify and prioritize the BIPs by their impact on compatibility, I wrote and submitted BIP123 which is now active and in use. This process BIP requires any BIP that affects compatibility to specify which of four different layers it affects: consensus, peer services, API/RPC, and applications.
All the layers except for the consensus layer afford introduction of new optional features and eventual deprecation of features that fall into disuse using techniques which pose essentially no risk of network partitioning or chain splits.
The consensus layer is by far the most difficult to change since it defines the rules determining whether a particular block (and thus, a particular blockchain) is valid. This includes checks for proper block data formatting, size limits, ensuring the block reward is not excessive, making sure all coins being spent actually exist, checking transaction signatures, and so on. Any change to these rules could create a chain split.
Consensus rule changes broadly fall into two categories: hard forks and soft forks. By definition, hard forks loosen or eliminate existing rules whereas soft forks only tighten or add new rules. It might not be immediately obvious but this distinction has some profound implications.
Since hard forks loosen or eliminate existing rules, blocks which would have been rejected by the old rules are now accepted by nodes using the new rules. Examples include increasing the block reward, increasing the maximum base block size, changing the block header format, or changing the proof of work function. Nodes that do not update their rules will reject these blocks and will continue to follow only chains that enforce the old rules. This means that unless all nodes update, we will get a chain split.
No hard fork has ever been done on Bitcoin to date. Or at least it is thought to be the case, since it is generally very difficult to ensure there are no undocumented behaviors that might result in effective loosening of rules. The closest we’ve come to this might have been the March 2013 chain fork caused by an undocumented behavior (or what some might call a bug) in a database library that caused some nodes to reject a block that should have been valid. This incident was resolved by explicitly rejecting this block and replacing the library in later versions of the software with one that (we hope) doesn’t have these sorts of issues.
Note that since only a rule was added during this incident — reject a particular block — the end result behaved like a soft fork rather than a hard fork. Nodes converged to a single chain, and now we’ll see why.
Soft forks only tighten or add new rules. Examples include reducing rewards, reducing maximum base block size, repurposing unused opcodes in certain ways to create new ones, or adding new data to blocks and committing the hash to an OP_RETURN transaction output that is ignored by older nodes. This last example is the mechanism used in SegWit, which we’ll get to later, to create additional block space without requiring a hard fork.
With soft forks, as opposed to hard forks, old nodes will continue to accept the new blocks since all the validation checks will still pass. Newer software will include new validation checks as well, meaning blocks that pass the validation checks of older software might fail on the newer software. The important implication for soft forks is that as long as a majority of hashpower enforces the new rules, all nodes will continue to converge on a single chain. This is emphatically not the case for hard forks.
Both hard forks and soft forks require a coordination effort to ensure smooth transition and to avoid or mitigate chain splits. Hard forks by default split the chain unless all the nodes on the network update their software by the time the rules change. Let that sink in…consider the logistical complications and philosophical issues regarding forcing all users (and application developers) to accept the change or the very real security risk of potentially causing a consensus failure.
Soft forks afford much smoother deployment mechanisms that do not require all nodes to update at once and provide many more mitigation mechanisms to avoid chain splits. Furthermore, the kinds of soft forks that have been deployed in Bitcoin have been specially designed to be backwards-compatible and preserve expected behavior of older software. This means you can spin up an old wallet you had forgotten about on some old hard drive and it will still work as expected! You’ll still be able to send and receive bitcoins the same way you always did.
The process of transitioning to new consensus rules is called activation.
Satoshi’s introduction of the 1MB block size limit back in September of 2010 is an example of a soft fork. It set a hard cutoff at a particular block height. Any block after that that exceeded the limit would be rejected. It was a very crude activation mechanism. But the network was still sufficiently small at the time and the stakes were sufficiently low that coordination was relatively simple and uncontroversial.
Later on, a new activation mechanism was employed that used a flag date rather than a block height. Two soft forks were deployed using this mechanism: BIP16 (P2SH), which allowed for multisignature transactions; and BIP30, which required all transactions to have a unique identifier.
Then came BIP34, requiring block height to be included in the coinbase transaction, which introduced a new activation mechanism, affectionately called ISM (IsSuperMajority) because of the name of the C++ function called to check whether it should activate. This was the first mechanism used that employed hashpower (or miner) activation, or what is now called a MASF (Miner Activated Soft Fork).
The basic premise was that the block version would get incremented by one with each such activation. Blocks with the new version would be required to enforce the additional rule once 75% of blocks within a 1000 block interval had the new version. Then once 95% of blocks had the higher version, blocks with the lower version number would be rejected.
This mechanism continued to be used for BIP65 (CHECKLOCKTIMEVERIFY) and BIP66 (Strict DER signatures). But significant limitations to this mechanism became apparent. Critically, since every soft fork deployment incremented the block version by one, soft forks could only be deployed one at a time — and until the previous soft fork activated, it was impossible to activate another one. If any activation failed for whatever reason, it would prevent any further soft forks from being deployed and activated.
This led to the development of BIP9 (Version bits), a soft fork activation mechanism that allowed parallel soft fork deployment and activation by using different bits in the version field for each soft fork rather than increasing the version number. Once activated, the bit would no longer need to be set and could be reused for a future activation. In principle, this would allow up to 29 soft forks to be deployed in parallel and activated independently. It still relied on hashpower signaling (by setting the version bit).
The reason IsSuperMajority and BIP9 relied on hashpower signaling is that even though soft forks do not require all nodes to update at once, miners should update to enforce the new rules to ensure they don’t end up mining blocks that will be rejected by nodes that enforce the new rules. Using miner signaling allowed for smooth coordination — as long as a supermajority of miners were enforcing the new rules, the risk of any chain split could be virtually eliminated. 95% was chosen just to be on the safe side. It was never intended to be a vote. The idea was that soft forks should be discussed and vetted prior to deployment to make sure they had the necessary ecosystem support. BIP9 was merely a solution to the transition coordination problem, avoiding disruptions to the network as much as possible.
The activation of BIP66, nonetheless, did not go entirely smoothly because even though miners were signaling for it, many were not actually enforcing the new rules. This led to some miners building atop a chain that was rejected by nodes that enforced BIP66. These blocks were subsequently rejected by the network and miners who mined atop this chain lost their rewards. This is known as validationless mining and is a generally discouraged practice.
BIP9 was first used to activate the BIP68/BIP112/BIP113 soft fork bundle which added the CHECKSEQUENCEVERIFY opcode allowing for relative timelocks. This deployment and activation went smoothly. Then came the second soft fork that was slated for activation using BIP9, BIP141 (SegWit), and this…is where things fell apart…and would never be the same again.
Despite having done a tremendous amount of outreach to users and industry, this particular soft fork became highly politicized as a result of misunderstandings as well as what seems to be a divergence of interests that was not openly discussed and acknowledged prior to its deployment. Specifically, despite being told by all the largest mining pool operators that they would signal for it, when it was actually released, a significant amount of hashpower refused to do so. A few mining pools began using threats of running other versions of the Bitcoin software that were incompatible at the consensus layer and making shifting demands, which to be quite frank, confused the hell out of most of the developers.
In hindsight, it’s become clear that miner activation of soft forks cannot be relied upon when there exists a divergence of interests between miners and users. In the cooperative case, it is a tried and tested mechanism which if done correctly is known to work smoothly. However, in the adversarial case it simply does not work. BIP9 effectively gives a small amount of hashpower veto power over any soft fork activation no matter how widely popular it is among users and industry.
Remember, the 95% was meant as a safety threshold, not as a vote. It was chosen arbitrarily. In truth, even something as low as 60% might have been sufficient to ensure no permanent chain split, although 95% tends to drastically reduce the orphan rate and the potential for a temporary chain split that persists for more than a few blocks.
And remember that before the advent of miner activation, Bitcoin relied on flag date and block height activation which depend on economically important nodes updating their software as the incentive for miners to follow suit. This type of mechanism has been since given the name of UASF (User Activated Soft Fork).
So given these lessons of the past, it seems that a combination of MASF and UASF will be required for future deployments to properly address these adversarial scenarios. It is unlikely BIP9 will be ever used again.
Unfortunately, BIP141 (SegWit) has already been deployed using the BIP9 mechanism and is as of this writing running on over 80% of nodes according to Luke-Jr’s charts. Redeployment using a new activation mechanism is a lengthy process fraught with complications. This means that user activation can only be realistically done on a short time scale by users running software that requires that miners signal for it using bit 1 as deployed with the BIP9 mechanism on nodes already out there.
This is what led to the development of BIP148 and BIP149 by Shaolin Fry. Both can be considered user activation mechanisms but are very distinct. BIP148 requires that miners signal for BIP141 starting on August 1st, 2017 until the activation locks in and is much more aggressive than BIP149, which waits until the current BIP9 deployment times out in November of 2017 and redeploys it using BIP8, a modification of BIP9 which still allows miners to signal but sets a hard deadline upon which it must activate. BIP149 could mean BIP141 (SegWit) doesn’t activate until well into 2018.
Please note that neither BIP148 nor BIP149 require that miners support SegWit transactions or include them in their blocks! They just make sure that miners cannot stop other miners from doing so.
Given the scaling pressures the network is experiencing and the tremendous advancement in payment channel technology that depends on BIP141 (SegWit) to function properly, I believe the current conflict between users and a few miners merits prompt resolution. BIP141 already activated successfully on other blockchains including Litecoin whose price rose significantly as a result of its activation.
By running BIP148 and insisting that the wallets and exchanges you use support it, it sets a hard deadline for activation. Miners must either activate in July or must signal starting August 1st until it activates. Miners will have to allow other miners to mine SegWit blocks and allow users to use SegWit-enabled software. If they don’t, their blocks will not be accepted by the economic majority and so they will not be able to sell any bitcoins they mine or collect as fees at the price that SegWit-supporting users and companies are willing to pay.
There are many possible scenarios that could result assuming a few miners continue to insist on blocking SegWit — but all of them are very costly for such miners, whereas mining on the SegWit-supporting chain will continue to be highly profitable. Since BIP148 is a soft fork, once it gathers sufficient hashpower the entire network will converge on it as the longest valid chain and the ecosystem can continue building atop the latest protocol innovations. We will finally get past the noise and carry on with the business of building the future!
