The Bitcoin Testnet3 Epoch 724 Blockstorm
The Bitcoin Testnet3 experienced a “blockstorm” tonight! While testnet blocks are supposed to mimic the main network and arrive on average every ten minutes ‡, thousands of blocks were mined tonight in the matter of a few hours.
In the past I’ve heard several plausible-sounding explanations for testnet blockstorms: miners attacking the testnet, a purposeful bug to prevent testnet BTC from becoming valuable, &etc.
I’ve never looked into it before, because it’s a testnet and the effect is only temporary. But this week I was the release manager for RADAR ION, a one-stop guide to joining the Bitcoin Lightning Network. ION lives on Bitcoin Testnet3 for the moment while we prepare our Lightning wallet and app suggestions for Mainnet, so I’ve been particularly interested in the testnet’s health.
The Testnet3 20 Minute Rule
Bitcoin Testnet3 has a feature that ensures the testnet is still able to mine blocks if its largest miners suddenly abandon it. From Bitcoin Wiki: Testnet,
Minimum difficulty of 1.0 on testnet is equal to difficulty of 0.5 on mainnet. This means that the mainnet-equivalent of any testnet difficulty is half the testnet difficulty. In addition, if no block has been found in 20 minutes, the difficulty automatically resets back to the minimum for a single block, after which it returns to its previous value.
Miners often test their powerful ASICs on Testnet3 before deploying them on the main chain. Testnet hashrate can fluctuate substantially, so this feature ensures that users never have to wait more than 20 minutes for a block, even if the only miner left on the network is running on somebody’s flip phone.
Normal Operation
A quick inspection of some recent blocks from the Testnet3 blockchain using the python-bitcoinlib library shows this feature in action:
import bitcoin
import bitcoin.rpc
from datetime import datetime as DT
bitcoin.SelectParams('testnet')
proxy = bitcoin.rpc.Proxy()for blocknum in range(1457000,1457010,1):
block = proxy.getblock(proxy.getblockhash(blocknum))
ntime = DT.utcfromtimestamp(block.nTime).isoformat()
print(f'block #{blocknum}:{ntime}, diff {block.difficulty}')
Output:
block #1457000: 2019–02–13T12:28:50, diff 14570951.397598663
block #1457001: 2019–02–13T12:46:41, diff 14570951.397598663
block #1457002: 2019–02–13T12:57:45, diff 14570951.397598663
block #1457003: 2019–02–13T13:01:01, diff 14570951.397598663
block #1457004: 2019–02–13T13:21:02, diff 1.0
block #1457005: 2019–02–13T13:28:02, diff 14570951.397598663
block #1457006: 2019–02–13T13:48:03, diff 1.0
block #1457007: 2019–02–13T14:08:06, diff 1.0
block #1457008: 2019–02–13T14:27:08, diff 14570951.397598663
block #1457009: 2019–02–13T14:43:44, diff 14570951.397598663When it’s been 20 minutes since a block, the difficulty for the next block is dropped to 1.0, and a block is solved almost instantly. On the very next block, the difficulty has returned to its previous value.
Blockstorm!
But something different happens in a blockstorm:
import bitcoin
import bitcoin.rpc
from datetime import datetime as DT
bitcoin.SelectParams(‘testnet’)
proxy = bitcoin.rpc.Proxy()for blocknum in range(1457563,1457573,1):
block = proxy.getblock(proxy.getblockhash(blocknum))
ntime = DT.utcfromtimestamp(block.nTime).isoformat()
print(f'block #{blocknum}:{ntime}, diff {block.difficulty}')
Output:
block #1457563: 2019-02-18T05:43:26, diff 14570951.397598663
block #1457564: 2019-02-18T05:47:15, diff 14570951.397598663
block #1457565: 2019-02-18T05:57:25, diff 14570951.397598663
block #1457566: 2019-02-18T06:08:26, diff 14570951.397598663
block #1457567: 2019-02-18T06:28:33, diff 1.0
block #1457568: 2019-02-18T06:28:32, diff 1.015794531589063
block #1457569: 2019-02-18T06:28:32, diff 1.015794531589063
block #1457570: 2019-02-18T06:28:32, diff 1.015794531589063
block #1457571: 2019-02-18T06:28:34, diff 1.015794531589063
block #1457572: 2019-02-18T06:28:34, diff 1.015794531589063The difficulty has gotten stuck near 1! What happened?
A Quirk in Difficulty Adjustment
Bitcoin is designed to keep blocks coming at a regular pace, even as miners change the amount of hashpower that they apply to the chain. This is accomplished by adjusting the difficulty parameter every 2016 blocks.
And here’s the bug: when the 20-minute rule kicks in for the block right before a difficulty adjustment and drops the difficulty to 1, the difficulty adjustment for the next 2016-block epoch is calculated against a difficulty of 1 instead of against the prevailing difficulty!
Block #1457568 marked the Bitcoin Testnet3’s 723rd difficulty adjustment, and therefore the beginning of its 724th block epoch. Block #1457567 triggered the 20 Minute Rule and temporarily dropped the difficulty to 1.
Instead of returning to the previous difficulty of 14,500,000, the faulty difficulty adjustment means that the difficulty is stuck near 1 for 2016 blocks. These blocks are solved in only a few minutes, but the difficulty algorithm can only adjust the difficulty by approximately a factor of four from one epoch to the next, no matter how far the chain is operating from the target 10-minute blocktime.
The chain difficulty must work its way up over several block epochs over the course of several days:
How Often Does This Happen?
Quite often:
What Are The Implications?
A few times a year, we get 3 to 8 days of unusually fast blocks. In the moments right after a blockstorm begins, several blocks may be mined per second.
Blockstorms provide a glimpse into what would happen if the Bitcoin block propagation rate was too low compared to the expected time between blocks. This could occur if the time between blocks was lower, network speeds were lower, or Bitcoin blocks were bigger.
A consortium of the best-connected miners—or even the single fastest mining entity—are at an advantage in this situation, because they can build on their own blocks and continue to extend the chain before blocks have time to propagate to and from less well-connected miners.
The result is extreme centralization of mining:
How Do We Fix It?
A fix would require a hard fork. Testnet3 miners and users—wallets, block explorers, faucets, testnet demo apps—would need to switch to a version of the Bitcoin software that fixes this quirk in the difficulty calculation. If some miners and users continue to use the old software, they would continue off on a different fork of the Testnet3 chain. This would require lots of coordination!
While tonight’s Testnet3 blockstorm was the first of 2019, it’s extremely unlikely to be its last. Testnet3 has been running since February 2011 and this bug is still with us 🐛.
‡ because of the 20 Minute Rule described above, the Bitcoin Testnet3 blocktime distribution is a truncated version of Bitcoin Mainnet’s (roughly) Poisson distribution.
