On Friday 9th August the Nexus network was subject to an attack allowing the attacker to generate blocks with an arbitrary coinbase reward.
At approximately 8am UTC the team were made aware of reports that some nodes on the network had dropped out of sync at block 2778365. We started our investigation and found that this was due to the error “block 2778366 ambassador signatures invalid”. This check makes sure that the ambassador rewards — which are included with each coinbase block — are paid to one of the 13 hard-coded ambassador keys. The error suggested that the block had simply been constructed with an invalid ambassador public key in the coinbase, but upon investigation of the data, this was not the case — the ambassador key in block 2778366 appeared to be valid.
By 10am UTC it became apparent that all nodes on the network running the Tritium wallet/core (version 3.0.x of the core) had dropped out of sync with this error, but nodes running the legacy wallet (version 2.5.5) were still in sync. Thinking that the newer Tritium code was at fault, we began debugging the Tritium code where this error is generated and found that the root cause was that block 2778366 had been created with block version 2, instead of the current version 6. This caused the Tritium code to compare the ambassador key signature from the block with an older set of ambassador keys that were in use before block version 6, leading to the invalid signatures error.
Once we had established that the block had been broadcast with the incorrect version we turned our attention to the legacy 2.5.5 code to understand why these nodes had not encountered the same error. We quickly realized that this older code skips the checking of the ambassador keys for blocks prior to version 3. That logic is correct and explained why the 2.5.5 nodes did not receive the same error as the 3.0.x nodes. However all nodes should have been rejecting blocks that are not on the current version (6).
Since it was apparent that blocks were still being mined and the chain continuing to move on 2.5.5 nodes, we begin checking all blocks mined after 2778366 to see if we could find any patterns. We established that all of the v2 blocks were mined on the Prime channel, and that they were frequent enough to suggest that this miner has significant mining power (hardware).
At approximately 12:30pm UTC we discovered that block 2778456 contained a coinbase reward of 497454.1 NXS, and that it had been confirmed by the network. Realizing the severity of this, we immediately contacted Binance and Bittrex to raise the alarm and suspend deposits and withdrawals. Binance did so immediately but it took Bittrex as much as another couple of hours to respond (actual suspension time unknown).
Further investigation of the 2.5.5 code revealed a logic error that would skip the checking of the coinbase reward values for block versions less than 3. It was this logic error that allowed the attacker to insert a coinbase reward of 497454.1 NXS and have it validated by other 2.5.5 nodes. The error was that all nodes should have been rejecting version 2 blocks, protecting against this.
At approx 14:30 UTC we identified that the code that should reject all blocks with versions older than the current version had a flaw, and would in fact only reject blocks of current — 1 (version 5). Version 2 blocks were being allowed to pass this check. It was this bug, in conjunction with knowledge of the code paths for checking the ambassador keys and PoW coinbase rewards, that allowed the attacker to exploit the 2.5.5 code and generate blocks with an arbitrary reward value.
In all we identified 3 blocks that contained an inflated block reward of approx 497k NXS reward — blocks 2778456, 2778527, and 2778553. Reviewing the times of the transactions, approximately 69k NXS were sent to 2R9nBJatGu82BrAwuuR8yxGrUH5BJmixBJqqhJN9wJSksM24jmL at approximately 12:00 UTC and another 69k sent to 2RAAc8KwJazE3FVZqQ9SdSjvvdQh4CFU6YFRM2ERVboHMf64YHJ around 10 minutes later, both prior to Binance suspending deposits.
It has since been confirmed that these deposits resulted in losses for Binance, as they were liquidated and withdrawn before the accounts could be frozen. To maintain a strong relationship with Binance, we honored their request to cover the outstanding amount of 145820.4164 NXS in the following two transactions:
We covered these losses from the remaining unclaimed funds left over from the April’s GitHub attack. The remaining balance is 82k NXS. Despite this, we will continue to honor fund-recovery requests, even in the case the remaining balance is depleted in which requests will come out of embassy funds. This decision was made due to limited embassy funding, time sensitivity for decisions to maintain good standing with Binance, and being that there have been little to no requests for fund recovery over the last few months.
You can monitor the fund recovery address here if you are inclined:
A further 855k NXS from the inflated blocks were sent to various addresses between 13:30 and 14:30 UTC. Fortunately, we confirmed that none of these addresses were Bittrex (which was still allowing deposits during that time), meaning Bittrex did not suffer any losses in this attack.
The team released a patched version of the legacy wallet (2.5.6), along with a planned release of the Tritium core (3.0.3 / 3.0.4), and the Tritium wallet (1.2.0) a little after 01:00 UTC on Saturday morning. These releases corrected the logic errors and reverted the updated nodes back to block 2778365, meaning that the first malicious block 2778366 would be correctly rejected. This essentially put all updated nodes on a separate fork of the chain to the attacker, resulting in a new chain without the malicious blocks or transactions. In short, the effects of the attack have been erased.
We urge all users to upgrade their nodes immediately, with Tritium recommended:
Tritium wallet: https://github.com/Nexusoft/NexusInterface/releases/tag/v1.2.0
Tritium core: https://github.com/Nexusoft/LLL-TAO/releases/tag/3.0.4
Due to the nature of this exploit affecting 2.5.X nodes, and this version of the software being no longer maintained, we DO NOT recommend running it. Use at your OWN risk.
QT wallet (deprecated): https://github.com/Nexusoft/Legacy/releases/tag/2.5.6
We are happy that even under such circumstances, we were able to mitigate the attack with minimal damages done to the network and overall community. The development team was quick to isolate, respond, and resolve the bug before it became a much bigger problem. A big thank you to everyone on the team for being on point, and keeping the network safe!