Back in 2015, when the issue of increasing transaction throughput was submitted for review, the framing was off. People had talked about the block size limit even since before Satoshi actually added the 1MB limit originally in 2010.
But the framing was off given all the new technological innovation that had taken place since then! (I’ll get back to this later)
I wrote in my previous piece that since 2010, a number of soft forks have been deployed successfully, employing a series of mechanisms that were increasing in sophistication. By the middle of 2015, there had been mastery of techniques for adding a bunch of new features at the Bitcoin consensus layer while preserving backwards compatibility and not disrupting the functioning of the network.
Why is backwards-compatibility important?
Anyone who has ever worked in deployment of new software products knows the logistical difficulties of getting everyone to upgrade. This is even the case assuming there are no reasonable technical or political objections to the update.
Whenever Microsoft releases a new version of Office, they make sure to also create tools to read older file formats and convert to newer ones. They know people will be using different versions at any given moment — some people still even use Windows XP (over 15 years old)! Despite Microsoft no longer officially supporting it they still released a fix for XP after the recent ransomware attacks such as WannaCry.
I bet most readers have had to endure the annoying nagging of their smartphone telling them it’s time to update their operating system, always wondering whether the update will break something…and having to wait for the whole process to take place, not being able to use the device for a while.
Blockchains are even more critical than operating systems in this regard. Everyone on the network must agree to the same consensus rules. Any change to consensus rules potentially subjects the system to critical failure modes.
We can continue hoping people update their software to something that continues to be supported…but it’s impossible to be entirely certain…and for highly security sensitive issues (such as securing tens of billions of dollars’ worth of other people’s assets) it’s of paramount importance that people be able to migrate easily, without the worry of potentially serious security threats.
Add to the logistical difficulties the potential for divergence of economic interests and you risk system attacks and chain splits. In most other kinds of systems, forks are not necessarily such a major problem…you need some competition and divergence in a system for it to evolve. But the consensus layer of the world’s longest continually running and most highly capitalized blockchain is probably not the ideal place to be fighting these battles.
Introduction of incompatibility in a system is sometimes necessary to improve upon it — but it also creates considerable friction. There is strong inertia around a set of practices and standards (even if ad-hoc). In the case of something like a word processor, it is always possible to add new features but also distribute tools that can convert old file formats to new ones.
Making a blockchain backwards-compatible
One of the big breakthroughs that was taking place around 2015 was the discovery of a fairly generalized method for adding new consensus features to the protocol while retaining backwards compatibility.
I went over the distinction between soft forks and hard forks and their implications in my previous piece.
For a long time, it had been believed that since soft forks can only make rules more strict or add new rules, the kinds of new features we might add to the protocol would be limited. Surely being able to loosen or even get rid of or change rules completely affords more extensibility, no?
Well, it turns out that there does exist a class of changes for which soft forks would not suffice — for instance certain fixes to the block header format or proof of work issues. But perhaps surprisingly, it is possible to add just about any new transaction feature without the need for a hard fork. And in many instances it’s actually practical to do.
The basic technique relies on committing to extra data by storing its hash (which is ignored by older nodes) somewhere in the block. And it turns out that SegWit was a prime example where this technique was a superb choice.
SegWit as great example of this technique
The reason Segwit was a superb choice of this technique was because it had become clear for a while that sticking the signature data directly into the transaction structure itself had been a mistake in the protocol’s original design. Not only is it generally a good idea to be able to process the signature (or witness) data separately — it should not contribute to the transaction ID.
Putting this signature data in a separate structure at once allows for more elaborate processing and extensibility of it (i.e. more elaborate smart contract scripts) as well as fixing malleability since the transaction ID no longer depends on it.
We can still include this data in the block by relaying it together with the transaction to nodes that accept it and committing to a hash of it in the block itself. And this has yet another benefit — we’ve created a new data structure that can be included in blocks while still retaining compatibility with nodes that check the limits of the older block structures.
Older nodes cannot process this extra data, so cannot perform all validation checks. They will just ignore the data and process blocks without it. But they can still perform all other validation checks that existed prior to SegWit. So they can continue to transact on the network — send and receive transactions — even if they don’t update.
What this also means is that this extra block data is not subject to the 1MB block size limit. We can store more data in blocks without breaking backwards-compatibility with older nodes. This allows blocks to be much larger than 1MB.
Here’s a 3.7MB block on the Bitcoin testnet done to test edge cases: https://testnet.smartbit.com.au/block/00000000000016a805a7c5d27c3cc0ecb6d51372e15919dfb49d24bd56ae0a8).
And not only does this mean we double capacity for typical transactions that take advantage of this feature —it also adds an incentive for cleaning up unspent outputs (UTXOs) which consume extra memory when running a validating node since spends require signatures which can now be handled more efficiently across the entire system and therefore are discounted when computing block totals.
So why the 2x?
Remember how I earlier mentioned that the framing was off?
It turns out that while top protocol engineers were working on these ideas back in 2015, a particular solution had been sold to a number of companies (some of which had surprisingly little contact with most of the top protocol developers); and to the public. A superficially simple looking change which just involved increasing the base block size limit, a “simple” change to a single parameter which results in massively much more complex dynamics than most people realize.
This was done outside of the peer review process and was not grounded on sound science.
While much easier to explain basic block size increases than something like SegWit, the full implications must be taken into consideration to understand why such a simplistic approach is highly problematic. Most critically, it breaks compatibility. And it does so in a way that pits some user interests against others, creating a lot of friction without having much long-term justification since blocks will just end up filling up again and fees will once again rise. It essentially amounts to kicking the can down the road but not really looking where you kick — and you happen to kick it into a hornet’s nest.
It is, in general, extremely hard to assess whether an incompatibility will create some perverse incentives or divergent interests or open up potential attack vectors that can lead to consensus failure or other serious security problems.
See, the thing with sophisticated attacks is you never see them coming until it’s too late. For the pro, the smallest of openings is enough to pry open a huge gaping security hole. People are trying to attack this network 24/7.
Again, we’re talking about a blockchain whose principal purpose is to secure what is now worth tens of billions of dollars of other people’s money!
Could we do a plain old base block size increase without consensus failure? Perhaps — given the right political and economic circumstances, it’s conceivable people could agree to a flag date and update software if done sufficiently in advance. But 2015 was just about the worst possible political environment for this type of thing. I don’t think the environment for this sort of thing is much better right now. But I’m hopeful it will eventually improve. However, I’m not ready to put a schedule on it and my advice is neither should you.
Had an actual base block size increase had the support of the economic consensus, someone might have just gone ahead and done it in what would effectively amount to a secession. We would get a chain split, but this would be the desired intent.
Despite people having done campaigns with rhetoric calling for a Bitcoin hard fork, not a single one of them to date has been actually carried out very far. This is even despite the fact that the entirety of the software needed to run a full validating node is open source and MIT licensed, meaning anyone can freely make modifications and release their own versions of it.
But for a few relatively minor incidents, no significant amount of hashpower has strayed from Bitcoin consensus rules in an attempt to hard fork because the reality is that their blocks will ultimately end up getting rejected by the economic majority. Exchanges wouldn’t accept them, wallets wouldn’t accept them, users wouldn’t accept them. They would be considered the equivalent of a counterfeit (or an altcoin).
Nonetheless, unfortunately, by late 2015 the framing had gotten so bad that people were just screaming for 2MB, period. It’s like in a sudden frenzy, the entire population around a nuclear power plant decides they need to double the size of its main reactor to keep up with demand — without considering any of the effects that might have on the proper functioning of the power plant. I mean, what could possibly go wrong, right?
People forgot about the stated objective and obsessed over a single technical parameter. And some people saw an opportunity for creating a wedge in the community and perhaps a chance to mount a coup or otherwise attack the network for their own gains. (Coming up with a few examples is left as an exercise for the reader.)
After many months of mailing list discussions throughout 2015 and two conferences (Scaling Bitcoin) open to the public, dedicated to the topic of scalability — after talking to all the best Bitcoin protocol experts in the world — it became clear that SegWit could effectively double block size and fix a bunch of other problems as well — without breaking backward-compatibility, and therefore without a serious risk of a chain split or consensus failure.
But it was hard to placate demands for a 2MB hard fork, whether the demands were genuine or simply negotiation tactics employed by people looking increase their direct influence over protocol development, sometimes to the detriment of the Bitcoin community as a whole.
So now what?
So by the end of 2015 it was pretty clear that SegWit was a good way forward. After considerable more testing and review, we’ve been waiting for SegWit to activate since fall of 2016. The network is ready for the transition. It has been extensively tested on multiple dedicated testnets, has been active on the Bitcoin testnet since May of 2016, and has successfully activated on other blockchains largely sharing source code with Bitcoin such as Litecoin.
It would avoid a chain split and all the branding issues that accompany that as well as more severe economic implications. Many companies are ready for it. But unfortunately, as mentioned in my previous post, it was deployed in a way that gave a small hashpower minority veto power. This was in line with the conservative way of thinking that if a deployment fails status quo is to be preferred.
But this way of thinking has been shattered by the ways in which this arbitrary veto power that developers gave miners has been abused to the point of extortion.
Most miners have been cooperative, sometimes misinformed, and have just gotten stuck in the middle of something they never asked for.
Reasonable requests from miners are always very welcome…but demands from a small handful of miners have gotten more and more ridiculous over time. There’s very good reason to believe some of these people are not being honest about their motivations. And it’s seriously hurting innovation and progress in Bitcoin — and also hurting the vast majority of miners who didn’t ask to be stuck in the middle of this.
If we cannot smo0thly activate a tried-and-tested upgrade that has gone phenomenally well on other blockchains such as Litecoin, there’s very little chance of smoothly doing what amounts to a much more contentious change — a hard fork that is so contentious nobody has had the guts to actually go through with it.
But at this point this activation nonsense is all just theater. It is literally impossible for miners to activate a hard fork without the near universal support of users. Users need to agree to change their software. And the soft fork signaling was an arbitrary system that was designed to give the ecosystem a smoother means of upgrading but was mistaken by a few miners as a naturally-given power which they can now use for leverage to gain other influence, much of it not even related to protocol development. It’s gotten totally ridiculous and needs to stop.
There’s no reason why we must deploy soft forks in this way in the future. Truth is miners mainly provide a service of securing blockchains and generally follow whatever blockchain is most profitable to mine. They get rewarded for this service, some them very well. Miners get to build chains, but users get to decide which ones are valuable to them.
We just hope that miners come to their senses and activate SegWit using BIP9 in July (whether under the guise of SegWit2x or not). And if not, we will start enforcing it ourselves as users on August 1st by running BIP148. If all goes well, we will see no chain split and Bitcoin will moon! Once SegWit activates, it will no longer be necessary to run BIP148.
Regardless of whether or not miners signal for SegWit2x, it’s ultimately up to users whether they accept this kind of change or not. Signaling for SegWit2x as a miner and actually running it are two completely different things. It is crucial that regardless of what happens, SegWit activates on the 80%+ nodes out there that support it.
Two months for serious consensus rule changes with little actual peer review doesn’t seem like a smart bet…but as always, I believe it is the right of everyone to run whatever software they want to run. After years of practically no downtime and phenomenal security and network stability across upgrades, I will continue to run Bitcoin Core once SegWit activates.