Get rid of those Zombie Channels!

If you have been operating a Lightning Network node for a while now, chances are you have some zombie channels attached to your node. Yuck! This is no good! You should take care of this. Time for you to do some channel hygiene.

Heads up! This article was written from the perspective of an LND user. While the described circumstances are a common challenge for all Lightning users and as such implementation-agnostic, the tools and commands mentioned here are LND specific and differ for other implementations like c-lightning.

What are zombie channels?

Zombie channel is a fuzzy notion for channels that are kind of dead, but still roaming around. To be precise: Channels cannot really be dead, only nodes can. Hence from your point of view, a zombie channel is a channel for which your remote node disappeared for such a long time, that you would consider it not to come back online anymore; a node that you would consider to be dead.

A zombie channel cut open

Why bother?

If a channel does not serve any purpose anymore, there is no upside in keeping it. However, there are several downsides:

• Your money is not liquid as it’s allocated to the unusable channel. Reallocating the funds takes time.
• You could use this money for another channel that actually works!
• This channel is a burden on the network. As long as your channel stays open, all peers of the network need to maintain information about that channel. Closing it will remove the channel from the collective memory and let it rest in peace. This is only true for public channels.
• If your own node goes belly up, you lose all the money you kept in zombie channels!

Bolters reacting to your question about techniques to get your money back

Why would you lose that money? Let’s say you actually did back up everything that the Lightning developers told you to (for LND users that’s the seed and the newest version of the file called channel.backup) before your node exploded. Great! That will help you to get back all the funds from nodes that cooperate with you. Without this cooperation you’ll be left with empty hands though (DYOR: Static Channel Backups).

So, if you can already assume beforehand, that a node won’t be cooperating with you in the endeavor to recover your funds, because it’s been shut down a long time ago, why not just close that channel?

But .. but .. Watchtowers!

Nope. That money is gone.

How to identify zombie channels

The problem with zombie channels is that they cannot clearly be identified, since one never knows whether a remote node is shutdown forever or only temporary. Focusing on the following metrics will help you to at least make adequate assumptions:

• Node is active or inactive: Whenever one of your remote nodes is being turned off, it informs your node that it’s becoming inactive. At startup, it informs your node that it’s active again. Don’t overestimate the significance of this metric for your zombie classification, as you cannot determine if a node is inactive for just 2 minutes or if it has been inactive for several months already.
• Last channel update: The last time this channel has been updated by your remote node. According to BOLT #7, you may consider a channel to be neglected if the last channel update was received more than two weeks ago.
• Public or private channels: This metric does not directly help you to identify zombie channels, but it helps you to weight the other metrics. If a channel is private, it was probably never intended to be online all the time, so keep that in mind when looking at metrics like the time of the last channel update, because you might want to be more tolerant with these channels.
• Local balance: This metric does not help you to identify zombie channels neither, but it shows you the severity of a potential loss, when your own node bites the dust.
• Disabled: Both nodes that maintain a channel together may tag a channel as disabled, independent of each-other. A node would do so, because it wants to declare a channel as temporarily unavailable or also permanently unavailable. Reasons for this may be a loss of connectivity or a preparation to an on-chain settlement. Note that your remote node’s disabled flag does not necessarily get broadcasted to you, but may just be communicated to peers that are trying to use that channel (spam protection). So your node might not be up-to-date and hence this metric could be misleading.
• Channel age: When has your channel been created? Might be helpful metadata in one way or another.
• Channel initiator: Just remember, the node that initiated the channel is also the node that has to pay the on-chain fees for the channel closing.

Don’t panic! Don’t just kill every channel that feels like a Zombie. Use the metrics!

Bash script for zombie channel identification

I’ve written a simple Bash script that should help you to get a quick overview of your potential zombie channels. Adapt the first variable TIME_GAP to define what channels you want to retrieve data for; The elapsed time since last received channel update is used as a treshold.

Install the tool as follows [Linux command line]:

$ wget -O zombieChannels.sh https://gist.githubusercontent.com/gcomte/013212cc5c2e3662f63ffce5c81829b0/raw/a0c4524248daa10984390b0e7c183eb9e8a6eb95/gistfile1.txt
$ sudo chmod +x zombieChannels.sh

Change the TIME_GAP variable to fit your needs and then run the script:

$ ./zombieChannels.sh

As a result you should receive a listing of all the channels that did not receive any channel update within the given TIME_GAP:

Using lndmanage

If you want to use a more sophisticated tool for your managing your channels, you may want to consider installing lndmanage. This python tool will deliver all kinds of information about your channels to you, so you can bring your channel hygiene to perfection!

Once you have lndmanage installed, you can retrieve the zombie channel data by typing:

$ lndmanage listchannels inactive

Improvements coming with LND 0.9.0

Software engineer Carla Kirk-Cohen who recently joined Lightning Labs, submitted a pull request this summer, that should make zombie channel identification much easier.

The new feature enables LND to track peer uptime by monitoring peer online/offline events. This way, LND can provide much more accurate information about the reliability of channels. Needless to point out, how helpful this information will be for us zombie hunters and for automating the process of zombie channel purging in the future.

Carla’s PR has already been merged and will be shipped as part of LND 0.9.0.

Happy Routing!

Now that your node is fresh and clean, use those freed sats to BUIDL some new channels and hopefully route some more payments! ⚡⚡

Thanks to @Snyke, @gugol, @actuallyCarlaKC and @bitromortac for helping me out with this!