Witnet Testnet 9.0b fork, as ~35k nodes join the network. Block counting paused, and scalability solutions coming shortly!
Published in
4 min readJun 23, 2020
TL:DR
- Testnet 9.0b successfully launched on Friday 19th June
- Block counting lasted for 1945 epochs (just over 24 hours), with up to 35k nodes joining the network and 1848 nodes mining at least one block
- At around #1945, the network forked due to scalability challenges, and block counting was paused
- Later this week, a reboot of Testnet 9.0c will take place, but blocks in 9.0c will NOT be included in the Block Counting Period and will NOT be rewarded with Mainnet tokens
- An update on the expected date for resuming block counting and date for the new Testnet 9.1 release will be announced early next week
- Feel free to take down / stop your nodes in the meantime
Remember: we’ll be allocating Mainnet WIT based on the percentage of (and not number of) blocks minted onto the blockchain by each identity during the Block Counting Period, so even if block counting is paused for a period of time, it’s paused for everyone, and the total WIT allocation across the Phase 2 Pool remains the same.
Testnet 9.0b: What Went Well
- Testnet 9.0b successfully launched on June 19 at 9AM UTC. The new bootstrapping mechanism — which had been implemented in under 20 hours after the false start the day before — worked as expected.
- Block counting for the Incentivized Testnet Program started immediately upon launch and continued on for 1945 protocol epochs (24h 31m), during which 1848 different Witnet nodes got their block proposals accepted into the chain.
- This has been the Testnet with by far the greatest adoption. It is difficult to calculate the exact number of nodes in such a widely distributed and decentralized network, but most estimates range between 25K and 35K nodes.
Testnet 9.0b: What Needs Improving
- 25K-35K nodes was (obviously) a number far greater than expected at this point. Testnet 9.0b was intended to prove and stress-test certain parts of the protocol, but not particularly scalability in terms of unique nodes proposing blocks.
- By protocol epoch #1945, many nodes in the network were not able to process the protocol messages in real-time and started to become unresponsive. Most of those faulty nodes likely shared one common trait: their operators tried to fit too many instances of the node software into a single piece of hardware or VPS.
- Under current protocol and network parameters, if a node perceives that more than 30% of its peers are not reliable enough, it immediately goes into a recovery mode in which it drops all peer connections, samples a fresh set of peer addresses and tries to re-synchronize to those. In other words, for a node to consider itself “synced”, it needs to perceive a 70% consensus among its peers.
- In this case, the block validations around protocol epoch #1945 were intensive enough to knock out around 30% of the nodes in the network. At that point, most of the nodes in the network went into the recovery procedure, but they were unable to reach the 70% consensus requirement, as other nodes in the same situation are not suitable for building that consensus.
- As a consequence, the network halted production of new blocks after block #1944, and it was decided that, as defined within the Terms and Conditions of the Witnet Testnet Incentive Program, block counting had to be paused, effective immediately.
Imminent Actions: Feel free to stop your nodes. We will shortly be rebooting the Testnet as 9.0c, without Block Counting
- The changes that will make it possible for the network to withstand this kind of scenario (a significant part of the network going down) will take some time to implement.
- In the meantime, feel free to stop or take down your nodes.
- A reboot of Testnet 9.0 (named 9.0c) will take place at short notice later this week. This is important for developers building different parts of the software ecosystem (wallets, block explorers, bots, etc.) to have a live network where they can test their creations.
- However, blocks in 9.0c will NOT be part of the Block Counting Period. That is, they will NOT be rewarded with Mainnet tokens. If the community drives the same volume of nodes into this provisional network, it will break again. Making releases and bootstrapping new networks has a clear overhead — the community cannot afford to do a reboot every couple of days.
Next Steps: Launching 9.1 and Resuming Block Counting
- A 9.1 upgrade containing the first batch of improvements to mitigate or solve the existing limitations will be released as soon as possible. The developers are making a huge and rapid effort to effectively prioritize and implement the changes that will have greatest impact to improve the performance, endurance and throughput of the network.
- Please follow the witnet-rust GitHub repo and the #dev-general Discord channel closely for learning more and participating in the discussions about the changes that are being made.
- Blocks in the 9.1 Testnet will be counted. That is, block counting will be resumed as soon as the 9.1 network is live.
- An update on the expected release date will go out early next week. Keep an eye on the announcement channels and your inbox. We appreciate your patience, and as always, feel free to reach out if you have any questions.
