5 Things I learned “Un-rekting” myself on Lightning Network

This is a follow-up to the video that I made, (Directly Below), about a month ago.
If you didn’t know, I was playing around with a preconfigured Raspberry Pi running LND. During one of my upgrades, my channel.db and hard drive got “borked”.

I have learned a significant amount in my recovery efforts. I would like to share 5 things I’ve learned while recovering funds from a corrupted channel.db.

But First
I truly appreciate the time and dedication from
Lightning Labs. Special thanks to Conner Fromknecht for really digging into this problem with me and doing the heavy lifting, while also thinking of ways to help other lightning users not have this same issue. The passion and dedication from the Lightning Labs team is A+!

5 things I learned while unrekting myself:

  1. Using tmux sessions.
    It was unbeknownst to me that using tmux was so popular when running LND. This was interesting as normally you will have long-lived processes run as daemons, (Background process), instead of having an interactive console running the process. This seems to be popular with c-lightning as well, given this Blockstream Article. I imagine in the future, when LND is out of beta, there will be an easier “daemon” option, like what we see with bitcoind. The name after all is “Lightning Network Daemon”. After my experiences, I would personally recommend running LND through tmux instead of something like nohup for personal use.
  2. BoltDB.
    I learned about BoltDB/Bolt, a pure Go key/value database, that stores its data in a single file, in our case channel.db. At first, I thought BoltDB was a custom database made for LND (hence “bolt” ), this was purely a coincidence. Bolt was actually archived on Github and deemed unmaintained about a year ago. There has been a new project, “bbolt”, that has now taken its place and maintained here. It seems that we see bbolt commonly referred to as just BoltDB. Bolt is lightweight, even lighter than SQLite, and is pretty much just a library that you use within your app. It’s intended to be used by a single process. This is contrary to most databases, as they are normally used to serve many clients. Bolt uses B+tree as its structure and supports fully serializable transactions. I believe choosing to use a database that supports serializable ACID transactions was the right choice, given we are dealing with channel funds. Any performance hit you might take using a serializable DB is probably well worth it. I could go into other nuances I learned about BoltDB, but I would like to make a final note: It is recommended that you run Bolt on a SSD. If you want to save money on cloud hosting, you might want to run your bitcoin node on HDD and then run LND on a smaller SSD. You could then connect to your bitcoin node through RPC with user/pass/ssh cert specified in your lnd.conf.
  3. Debug levels
    Adjusting the debug levels made diagnosing the situation much easier. Setting these debug levels below helped give clarity to my situation:
    You can set these in your lnd.conf or at anytime with the lncli while LND is running.
  4. Running LND in Docker.
    One issue that could have been the cause for my situation was LND failing to catch all kill commands (Specifically ones produced by docker). As I’m not certain this was the cause of my problems or ungraceful shutdowns, it seems fairly plausible. I believe this potential issue has been fixed and has been merged into master for a couple weeks now. 
    Jameson Lopp gives a great presentation on “Riding the Lightning — Casa Node Challenges” and points out some other issues that they had to tackle running LND in docker. Presentation (around 4m 50s speaks on docker issues).
  5. Finally, lightning recovery.
    I have created a recovery repo to help other people who run across the same issue of a corrupt channel.db. Please read the README in detail. Your mileage may vary. Hopefully, you will never have to use this hacky repo, as improvements to LND and supporting software are happening everyday.

Also, I recommend reading over lnd/pull/2313, which covers SCB (Static Channel Backups).

By “fully settled”, we mean funds that are in the base commitment outputs, and not HTLCs.

To my knowledge, there is currently no official way to backup the funds in a HTLC for LND.

This infrastructure will be built out in the near future, but until then we have this scheme which will also be a fall back in the scenario that any higher level mechanisms fail.

Security and safety of funds is of paramount importance. The responsibility to keep your bitcoin safe has always been your own responsibility, lightning network is no different. Even though I’m sure that there will be some more bumps down the “road of progress”, I’m very optimistic about the future of lightning network. Until robust backup solutions have been created and tested, I inject a note of caution to anyone wanting to put a significant amount of bitcoin on lightning network.

See something erroneous? 
Please leave a comment to fix anything incorrect!

Michael Tidwell