Istanbul’s Rough Ropsten Launch: 5 Takeaways

Today we’re going to focus on the fairly messy launch of Istanbul onto its first testnet — Ropsten.

Pictured: if Ropsten had a break room. Photo by Nathan Dumlao on Unsplash

The next major upgrade to the Ethereum Blockchain is called Istanbul. Istanbul introduces several changes, ranging from modifying the gas price of certain operations, to introducing cryptographic functions that were previously unavailable. If you want the details about what exactly is going into Istanbul, there are better articles for that.

Today we’re going to focus on the fairly messy launch of Istanbul onto its first testnet—Ropsten.

Background

For a bit of background, there are four major Ethereum testnets. Ropsten is a proof-of-work testnet just like the Ethereum mainnet, with the only fundamental difference being that nobody considers Ropsten ETH to have any market value, and because economic value of the native coin is fundamental to the security of a proof-of-work chain, Ropsten isn’t very secure. The other three testnets—Kovan, Rinkeby, and Goerli—are proof-of-authority chains, meaning they are secured by trusted signers rather than computational power. The trade-off here is that those authorities could potentially collude to censor transactions or potentially rollback blocks, but so long as they don’t do that things run pretty smoothly. Neither of these models are secure enough to handle significant economic transactions, but both have their pros and cons for testing blockchain development. In general at Rivet, we recommend that Ropsten be considered the testnet for protocol developers, while Goerli and Rinkeby are better testnets for smart contract developers.

Preparation

The way hard forks are applied is that the core developers (or the maintainers of a given testnet) pick a block where the hard fork rules will start to apply. For Istanbul on Ropsten, the block 6,485,846 was expected to happen on October 2. These block numbers were decided and published on September 25th.

In the days that followed, the major Ethereum clients released updates. The versions supporting Istanbul included:

• Geth 1.9.5
• Parity v2.6.4-beta
• Besu (Previously known as Pantheon) 1.2.4
• Nethermind 1.0.8
• Trinity v0.1.0-alpha.29

Each of these clients had to make a new release after the block numbers, and end users needed to update their client before block 6,485,846 occurs or they’ll find themselves on the wrong side of the Istanbul hard fork.

At rivet.cloud we updated to Geth 1.9.5 within a couple of days of its release, so we were prepared for the early arrival of the Istanbul block.

The Fork

The publications talking about when the Istanbul hard fork was happening on Ropsten mostly focused on the October 2 date, but the trigger was a specific block number, not the date. As noted earlier, Ropsten is a proof-of-work chain without economic value on its token, which means the frequency of blocks can vary wildly. A substantial increase in hashing power lead to block 6,485,846 arriving nearly three days early, and many node operators and miners who had planned to upgrade before October 2 were caught by surprise.

The result was messy. Very little of Ropsten’s hashing power switched to Istanbul, so the pre-Istanbul kept going and quickly became the longer chain. The Geth team quickly added to Ropsten’s hashing power to get the Istanbul chain moving, but a day after the fork the non-Istanbul chain is still about 10,000 blocks longer than the Istanbul chain.

Even some major infrastructure providers were caught by surprise. Infura, who had already tweeted to assure their customers they were prepared for Istanbul on Ropsten, was on the non-Istanbul chain until about 14 hours after the fork landed.

Presently, if you’re trying to fast sync a Geth node with 1.9.5, you’ll need to add the flag

--whitelist=6485846=0x43f0cd1e5b1f9c4d5cda26c240b59ee4f1b510d0a185aa8fd476d091b0097a80

to tell Geth which block to take for the hard fork block.

Lessons Learned

There are several takeaways from the shaky launch of Istanbul on Ropsten.

1. Hard forks can come early (or late). Planning for a Wednesday gives wiggle room in both directions, but even then it’s best for node operators to be prepared for the fork the week before.
2. It takes a while for end users to upgrade. A big announcement about five days before the hard fork block hits puts a lot of pressure on users to upgrade. The teams running these nodes have existing priorities, and depending on their infrastructure it may not be as simple as dropping in a binary and restarting. Giving these teams a full two week sprint lets most teams slot it in around other priorities instead of having to rush to re-prioritize it.
3. Keep an eye on the client versions of the peers on the network prior to the upgrade. The admin_peers call returns information about all of your connected peers, which includes the client versions. This can help inform what percentage of participants in the network are prepared for the hard fork. If the percentage isn’t high, take to social media to spread the word.
4. Have mining power ready. Ask for volunteers with mining equipment to be ready to start moving the testnet forward on the right side of the hard fork.
5. Don’t rely heavily on Ropsten for your applications. Ropsten is generally the first of the major testnets to apply a hard fork, and also the most variable because of how fluctuations in hashing power can change the timing of the fork. If you need a low stakes network to operate on, lean towards Goerli or Rinkeby unless you have a low level dependency on proof-of-work properties of the chain.

Final Notes

In general, we can expect the forks for Goerli, Rinkeby, Kovan, and Mainnet to go a bit more smoothly. The hard fork coordinators have contact information for all of the authorities on a proof-of-authority network, and can make sure they’re updated in advance of the fork, preventing a chain split. Additionally, many node operators will have updated in preparation for the Ropsten fork, and will hopefully have applied the updates to nodes for the other networks at the same time. For mainnet, the dates are generally determined much further in advance, and a lot more attention is paid to the percentage of nodes and miners that are prepared for the fork.