Hard Fork: CryptoNight Variant 2 Implemented

Context

The purpose of this post is to detail the blockchain attacks and the chain split that occurred over the past six days.

TL;DR: Lethean hard forked, meaning a software update is required to continue using the blockchain. New downloads are available now for the CLI and GUI. Exchanges have been informed of the update and should resume deposits/withdraws soon.

Blockchain attack

On October 20 at approximately 21:00 UTC, an attack was attempted on the Lethean (LTHN) blockchain. The attacker attempted to exploit the difficulty algorithm, which is the portion of the blockchain code that decides how much work is needed to make a block. Ultimately, this initial attempt failed, and the attacker later attempted a secondary attack on the blockchain around October 21 00:45 UTC. The successful attack operationalized the same flaw in the difficulty algorithm, allowing several thousand blocks to be mined with ease.

Exchanges During Blockchain attack

When it became clear to us that the blockchain was potentially under attack, we immediately reached out to exchanges and requested to have deposits and withdrawals stopped. Fortunately, the timing of our response meant that no exchange was able to receive deposits and withdrawals after the attack. This was confirmed by both exchanges currently listing LTHN (TradeOgre, Stex).

After the attacks were recognized, our blockchain team went to work right away to understand the issue and create a solution. As it was determined that exchanges did not suffer any losses in the attacked blocks, the logical choice was to roll back the blockchain and prevent the attacker from receiving any of the coins they mined. More information about the initial response and our strategy can be found in our October 22 attack post.

Hard Fork

On October 23 at approximately 15:00 UTC, we finalized our hard fork and released 3.0.0.b3 (“beta 3”). This hard fork implemented CryptoNight variant 2 proof-of-work into our blockchain, following Monero, and fixed the attack method used on October 20. Although we performed extensive testing on “beta 3” release before pushing it to users, the described bug did not reveal itself until running on the network for many hours. After the beta 3 release, a bug was identified in some daemons.

Users that resynced the entire blockchain before upgrading to b3 experienced a difficulty bug that led to several chain splits and issues syncing on October 24. The bug took some time to track down and resolve, and ultimately lead to the release of 3.0.0.b4 (“beta 4”) on October 25. We have carefully observed the behavior of the beta 4 release over the last 24 hours and feel confident in its readiness for full release. We appreciate your patience while we monitored blockchain performance and waited to announce the release until it was ready. Accordingly, until this moment, we have kept exchange deposits and withdraws paused out of an abundance of caution*.

Software Update Instructions

Anyone with wallet version 3.0.0.b3 or lower will need to update to 3.0.0.b4: (GUI download) (CLI download). Once you have downloaded and updated your wallet, you will need to ensure that you have not synced past the current block height on the block explorer. You can cross check the current top height at our remote node.

After you upgrade your Lethean daemon and/or wallet, it is critical you first verify that your blockchain height matches the one displayed in the explorer and remote node. If your blockchain height matches and you either never downloaded beta 3 or never resynced your chain before beta 3, your upgrade is complete. If you used beta 3 and resynced your chain, it is possible your blockchain is corrupt.

Extended Update Instructions

The aforementioned corruption in the blockchain led to a splitting of the chain for some pools. We have contacted all known pool operators with update instructions to ensure they have resolved the issue.

To determine if your blockchain is corrupt, open the Lethean daemon (letheand) and run the command `print_block 260000`. Information for the block will print out; we are only interested in the `difficulty` portion. The difficulty for that block should be 311321599 exactly. If it is anything different, your blockchain is corrupt. Two methods for remediation exist:

1) Run lethean-blockchain-import — pop-blocks <TOP HEIGHT MINUS 166000> to reduce your block height to 166000. The issue with diff occurred at block 166133, so popping that many blocks will allow you to sync successfully with proper difficulty. This command can take some time to run, so be patient.
2) Delete your entire database and resync from 0. This is more time consuming than #1.

After you’ve removed enough blocks to get yourself to the proper height, simply open your daemon and it should sync properly. It will most likely take a few hours to sync to the top height. Speed up syncing by adding the parameter ` — block-sync-size 250` when launching `letheand`.

If you experience issues syncing, you can use some or all of the exclusive nodes known to be correctly functioning: 69.162.83.203, 140.82.9.90, 192.124.18.154

Add an exclusive node to your daemon by launching it via command line: “letheand — add-exclusive-node 69.162.83.203”

Added fixes during the hard fork

• Difficulty algorithm upgrade to LWMA-3 including bug fixing of exploit in previous algorithm
• Reduction of the median time past (MTP) window to keep block times more consistent between nodes and reduce the potential of a future attack
• Enforcing limits on ‘old’ blocks entering the chain (courtesy of Masari)
• A couple upstream fixes from Monero involving transaction and timestamp processing

For all changes included in this release, please see the changelog.

Exchanges During the Chain Split

*After the initial beta 3 release and prior to the chain split issue arising, we contacted TradeOgre and Stocks.Exchange (Stex) and requested that they reopen trading. This occurred on October 23 21:02 UTC. TradeOgre reopened deposits and withdrawals at 23:34 UTC. Once it was clear that our chain had forked, we requested that deposits and withdrawals be paused again on October 24 11:52 UTC. Since Trade Ogre had placed a 99 confirmation hold on deposits and withdrawals, deposits and withdrawals from the time the chain split until the time the pause was requested did not enter or leave the exchange. The chain split was recognized and managed quickly enough that adverse effects were minimized. Stocks.Exchange deposits remained closed during the entire “beta 3” chain split issue.

Mining pools

At this time, miners should not be concerned with using the following preferred pools which have been confirmed to be properly updated**:
https://lethean.io/pool
https://lethean.blockharbor.net
https://pool.letheancoin.com
https://lethean.west-pool.org
https://lethean.hashvault.pro/

Summary

On October 20 at 21:00 UTC, an attack attempt was made on our blockchain. This initial attempt was unsuccessful. On October 21st at 00:45 UTC, a successful attack was made on the blockchain. This attack required us to hard fork, roll back the blockchain, pause exchanges, and implement a new difficulty algorithm. The team completed these tasks in a little over 36 hours.

After finalizing the hard fork, any person or pool that synchronized the blockchain from 0 or popped blocks off of their blockchain data and resynchronized from a point prior to or during the Lethean version 3 or Lethean version 4 block change (block 166133) had their blockchain incorrectly calculating the difficulty for new blocks. This resulted in pools splitting the blockchain.

During this time, exchanges were asked to pause deposits and withdrawals. This was requested to ensure that their internal accounting remained correct while still allowing users to trade Lethean. Since no new Lethean has entered or exited exchanges during this time, no one trading Lethean on either exchange should be impacted.

Final thoughts

We would like to take this opportunity to say thank you to both our community and our team. Although this was a chaotic time for our development team, we could not be more impressed with their work and diligence. We appreciate all of the community support that we received during this time, and we hope that we were able to make this hard fork as smooth as possible for you. We also appreciate the cooperation from pool admins as software had to be updated, blockchains manually fixed, and balances adjusted.

As always, lessons were learned, and we plan to perform better in the face of another threat to the blockchain. We know this post was quite dense in terms of information but we believe in transparency, and we think it is important for our community to precisely understand the events as they unfolded.

Throughout the course of the attack, we coordinated with several other coin developers to notify them of the difficulty algorithm exploit. We feel this type of communication is imperative in cryptocurrency and we hope to receive the same courtesy from other developers. We are exploring expansion of our Zabbix monitoring to receive more proactive alerts about impending attacks of this nature in the future, and considering further modification to the daemon code. We are in the process of contacting some cryptocurrency personalities for an in-depth interview and walkthrough on the attack, our response and the aftermath. If you’re interested in talking to us, please let us know!

We look forward to discussing all of our successes with you on November 1 for our regularly scheduled biweekly update.

Regards,

The Lethean Team

** If you are a Lethean pool admin and your pool is not listed here, please contact us on Discord.