Bitcoin Node Security Case Study (LND)

This story describes real events that just happened. Imagine you have a Zeus wallet in your phone to be able to pay with Lightning from your own Bitcoin node running Umbrel, RaspiBolt or similar. Suddenly, you realize your phone is stolen. After calling the police, you rush home, check your Telegram on the laptop, and see a message from BOS:

⛓ (pending) Sent 4,800,330. Paid 330 fee. Sent to bc1qey9my...

The thief has used Zeus to withdraw funds from your node! What to do?

Damage control

The ⛓ symbol in the message indicates that your sats are being stolen via blockchain. If it did not follow with the same text without the “(pending)” prefix, it means the transaction is still pending in mempool and has not been mined. However, your first priority right now is to prevent more funds from being stolen. There are plenty of those in your lighting channels. Pull the ethernet cable from your node and connect a monitor and a keyboard instead!

Now, open Terminal and stop your LND. This is how to do it on Umbrel, for example:

$ /home/umbrel/umbrel/scripts/app compose lightning stop lnd

For LND implementations without Docker this is simpler:

$ sudo service lnd stop

You should learn how to do it for your node, as well as where the lnd config folder is located. Go into that folder:

Umbrel: $ cd /home/umbrel/umbrel/app-data/lightning/data/lnd
Others: $ cd .lnd

Now you need to remove macaroon authentication by deleting the respective files. LND will create new, different, ones when it restarts (it is safe to plug in the internet cable now):

$ rm data/chain/bitcoin/mainnet/*.macaroon
$ rm data/chain/bitcoin/mainnet/macaroons.db

Umbrel: $ /home/umbrel/umbrel/scripts/app compose lightning start lnd
Others: $ sudo service lnd start

Now, let’s have a look at the stolen funds transaction. Maybe it is still possible to revert it.

Blockchain forensics

Copy-paste the address from the Telegram message into mempool.space search field and click on its last (or only) transaction id:

You see that it is still unconfirmed. Compare its fee rate (2.01 sats/vB) to what is currently required by miners:

Now we know the reason for the delay. The thief used the default rate in the Zeus wallet and did not look at the mempool situation before sending the payment. She was in a hurry! Three things may happen next:

1. Mempool returns to normal and the transaction gets mined.
2. The recipient of the stolen funds bumps the fee via CPFP and the transaction gets mined.
3. Enough time passes and the transaction gets purged from the mempool because the fee is below the cutoff 4.62 sat/vB.

Since the transaction still has not been fee bumped, there is hope that the thief does not know how to do it. This buys us some time. We will try to broadcast a transaction that spends the same output, but pays a higher fee rate.

The double spending trick

It is normally not possible to spend the same UTXO as an already pending transaction. LND prohibits it by “locking” the outputs. But we can make it forget all on-chain payments, including the one it just broadcasted, by telling it to rescan the wallet from the beginning — the date when it was created.

Bitcoin blockchain does not know about pending transactions in the mempool, so LND will forget about the stolen funds and let us send another payment to ourselves from the same UTXO. If we pay a higher fee rate than the thief, our replacement transaction will be mined first, invalidating the withdrawal.

The process of rescanning the wallet will take a long time — a few hours on a Raspbery Pi for a reasonably old wallet. Your node may be slow to respond or outright unreachable by peers. If you have any pending HTLCs with near expiration time, they may force close the channels. To prevent this, use this command in a separate Terminal/SSH session (it will continue running):

$ bos limit-forwarding --disable-forwards

In the original terminal verify that/wait until there are no HTLCs in flight:

$ lncli listpayments --include_incomplete | grep IN_FLIGHT

Make a backup of your lnd.conf and open the editor:

$ cp lnd.conf lnd.conf.bak
$ nano lnd.conf

Now add the following string as the first line, then press Ctrl-S & Ctrl-X to save and exit:

reset-wallet-transactions=true

For a double spend trick to work, bitcoin.conf should have the following flag enabled:

mempoolfullrbf=1

This is already so on Umbrel. Check your implementation and restart bitcoind if you had to add/change this flag:

$ nano .bitcoin/bitcoin.conf
$ sudo service bitcoind restart

Restart LND for reset-wallet-transactions=true to take effect:

Umbrel: $ /home/umbrel/umbrel/scripts/app compose lightning restart lnd
Others: $ sudo service lnd restart

Before you forget, restore lnd.conf to prevent the wallet rescanning on each restart:

$ cp lnd.conf.bak lnd.conf

Monitor the progress of the rescan by watching the LND log:

Umbrel: $ docker logs lightning_lnd_1 --follow | grep LNWL
Others: $ journalctl -n100 -fu lnd | grep LNWL

...[INF] LNWL: Opened wallet
...[INF] LNWL: Dropping btcwallet transaction history
...[INF] LNWL: Opened wallet
...[INF] LNWL: Started listening for bitcoind block notifications via ZMQ on 127.0.0.1:28332
...
...[INF] LNWL: Done catching up block hashes
...[INF] LNWL: Finished rescan for 21 addresses (synced to block ...

While waiting for the rescan to complete, open a notebook and prepare your replacement transaction:

bos fund <Own Addr> 10000 --fee-rate 100 --utxo <UTXO>:<Output>

<Own Addr> is one of your node wallet’s addresses. You may use the address from which the funds were stolen. But the best practice is to always generate a new address with:

$ lncli newaddress p2wph

<UTXO> is the previous confirmed tx that funded the withdrawal address. You can find it by clicking on the input address (left side) in the theft transaction. Note the amount of that input (0.0998009) and find the same amount in one of the previous transactions. The <Output> number is 0 in our case.

Wait for the wallet scan to complete (the log should say “Finished rescan”), verify that the fee is high enough (via mempool) and run your command:

$ bos fund bc1q6e4...qryz 10000 --fee-rate 100 --utxo cfe9baa..eaa92:0

send_to:
  -
    bc1q6e4u80dk9df...ncdqryz: 0.00010000
requested_fee_rate: 100

change:         0.09954709
sum_of_outputs: 0.09964709
spending_utxos:
  - cfe9baad14452732d27d5a...92e26da17e35a60353cf8eaa92:0

fee_tokens_per_vbyte: 100.00
signed_transaction:   0200000000010192b...10dd97dc113ffd3b00000000

It will not broadcast automatically! Copy-paste the signed transaction raw hex to broadcast it directly from Mempool.space:

The result for the same address now looks like this:

The thief’s withdrawal tx has been replaced:

…and our tx got mined a few minutes later:

But our job is not done yet. We want to check all of the UTXOs in the wallet in case the Telegram bot did not catch all the withdrawals. Run this command:

$ bos utxos

utxos:
  -
    outpoint:            70edb5b7a4102015e5b47...35907fa7c4:1
    amount:              0.04699835
    confirmations:       9
    address:             bc1qmtnm43j83d286cult..snrt3nwmwwq7mwqqf
    related_description: bos open
 ....

Put each outpoint in mempool.space and see if there are any unconfirmed transactions spending this output. Repeat the steps above if you find any. You may provide several utxo parameters to bos fund at once:

Prevention

The moral of the story is to never connect a phone app directly to your Raspberry Pi. The best practice is to run a separate node with a small private channel opened to your main node. So that the amount that can be stolen is limited to that channel’s balance.

Such a node can run right on your phone inside the Blixt wallet. You can still keep Zeus installed, but only connect it to your node via local network (i.e. umbrel.local) in order to refill sats to Blixt when you are at home. This will be like setting a withdrawal limit on your debit card. Stay safe!

PS. I was not the victim of this theft attempt. I watched the story evolving in real time and learned the necessary steps from Alex Bosworth and other hackers in his private BOS Telegram channel. I invite you all to subscribe to it as well. If you decide to tip the author, please send some sats here: vladgoryachev@ln.tips Thank you in advance!

PPS. For those wondering, the story happened in Switzerland and the thief was a hot russian girl!