Incident report: v17.0 local block rollback events and Binance incorrect deposits

The following report details the two recent incidents of local block rollbacks on a small set of nodes. The incidents were made possible by the introduction of epoch blocks and resolved with the latest V17.1 release.

Incident 1

On January 14th at 11 pm GMT an invalid fork resolution attempt was detected between a distributed account-open epoch block and a send block, with this root block. It occurred on one of our internal nodes and on a handful of other nodes on the network, most notably the Canoe and Nanowat.ch representatives.

During propagation of the epoch block across the network, the affected nodes went through a series of elections and rollback events related to invalid matching roots on the blocks (see root cause analysis below for more details on this). This caused a rollback of related blocks on the send block’s account to its predecessor from ~15.8 million to ~11.9 million, totaling approximately 3.9 million blocks (see chart below). Due to the design of the network, the few nodes affected stayed connected and were able to fully resync between 12–20 hours later without interference or updates.

Incident 2

On February 5th at 12 pm GMT we received multiple reports of unexpected deposits of Nano into users’ Binance accounts. We immediately requested Binance suspend deposits, withdrawals and trading, and later notified the community while working with Binance to diagnose the issue. It was discovered that older transactions were received again on their node during the process of resyncing. This issue was possible in V17.0 of the Nano node they were running and was triggered similar to the epoch block distribution issue mentioned above.

Due to their particular wallet implementation, these old transactions were incorrectly credited to user accounts in their system. Binance has upgraded their nodes to V17.1, initially released on January 21st which resolves the epoch block related rollback issue, and is working to resolve the incorrect deposits that occurred on user accounts. We are also in the process of assisting them in a review and update of their wallet implementation of Nano. Normal operations for Binance will resume once the situation is fully resolved.

Root cause analysis of the block rollbacks

After thorough code reviews, we identified an issue where the node would incorrectly determine that two blocks conflicted. The node detects conflicts between blocks by comparing the block’s “root” for equality. For non-open blocks, its root is the hash of the previous block (block hash root). For account-open blocks, the root is the account numbers public key hash (account number root).

The conflict detection code relies on the impossibility of having a block hash equal an account number’s public key hash. This impossibility exists because block hashes are random and it would not be possible to produce a public key hash for a chosen account number by the account signer since this would require deriving a private key from a public key. When epoch blocks were introduced, this impossibility no longer applied because an account can be opened by a signer other than the account signer, which is the epoch signer.

So the following conditions must be in place for this issue to have the potential to occur:

  1. The issue can only happen when epoch-signer account-open epoch blocks are published; it cannot happen through normal network operations.
  2. This issue could not have happened before the commit on 2018–07–14 when epoch blocks were introduced, so only nodes V15.0 to V17.0 had the potential to be impacted.
  3. The account-open epoch block would have to have its root value match another block hash in the ledger.

In addition to these conditions, a specific series of events had to occur to cause the incidents: some nodes would have to receive the epoch block before a majority of the network had seen it. Within this timeframe it must have sent a query out to at least one node that had not yet received the epoch block which would then have responded with the old block it thought was in conflict and this would have been added to the local election. Once both were in the election and the epoch block received sufficient quorum, it would initiate a rollback of the old block thinking this was necessary to fit the epoch block. Since this rollback was actually unnecessary as there is no conflict, subsequent bootstrapping would have inserted the old block again and the node would recover.

Resolution

V17.1 was released to resolve the issue, preventing the node from seeing conflicts between roots that are account number public key hashes and roots that are block hashes. We have been in contact with Binance and other services and exchanges to verify they have upgraded their nodes to this latest version.

It’s important to note: no double spend occurred on the network and the issue was not caused by the 17.0 to 17.1 upgrade.

We want to provide confidence that in the future any key upgrades to the network will happen as broadly and quickly as possible, so we are reviewing our processes to ensure that we have streamlined communications in place for notifying and verifying exchanges and services of these releases going forward.

We will also be following up with additional details and best practices around integrating Nano with exchanges and services in the coming weeks.

Thank you for your patience as we’ve worked through the V17.1 release and upgrade process. We are currently testing our next Dolphin release and look forward to continuing improving the Nano network for everyone.