The Art of Routing Sats

With the public release of LND 0.18 node runners gained extra flexibility in policy settings. Let’s discuss today’s best practices and tools for automated fee management.

No More Mr. Nice Guy

Running a Bitcoin Lightning node is an economic activity like any other. The main principle of capitalism is to seek maximum return on the employed assets. For a channel such an asset is its local balance. Ignoring the hardware and electricity, the cost base of a channel equals chain fees to open and close it. While the former can be minimized by boosting the size, timing Mempool and batching the opens, the latter is unpredictable and potentially very high due to a chance of forced close. Even channels opened by remote peers will cost you dearly to sweep and to redeploy the fragmented UTXOs.

You can reduce this risk somewhat by not opening to and not accepting channels from nodes with limited size and history (with bos inbound-channel-rules). Another must have is a Clearnet connection. Tor entry nodes often throttle lightning traffic and may fail you at a critical moment. Use Tunnelsats or configure your own VPN. Make sure to limit the number of concurrent HTLCs with Circuit Breaker, so that when a channel does fail the number of sweeps will be limited. Have a UPS and a reliable ISP, and boast to the world how stable your node is with Amboss Healthcheck Status.

But even without an FC a channel cannot route forever. Because of how LN operates, the channel database size is a limited resource. Each non-dust sized forward creates a penalty transaction, and all of them must be stored for the life of the channel. Raspi-based nodes struggle when the database grows over 3–4 Gb, so the routing frequency must be moderated, or channels constantly closed and reopened. Some nodes choose to only route decent size payments by setting Min HTLC size of each channel, to make it worthwhile. However, this filters out micropayments below dust size that do not grow the database and are very profitable with standard base fee of 1 sat. A better approach is to set higher outbound fee rates.

The days of Zero Fee Routing are over. Multiple waves of high Mempool devastated such altruists. Satograms and sat streams dented the enthusiasm of most Zero Base Fee cultists, because renting out your CPU and bandwidth for free is annoying. The need to moderate traffic for the database size control goes hand-in-hand with the need to recover chain costs. A smart, and automated, fee management strategy is necessary. You want to be greedy and keep selling your local liquidity to the highest bidder, while keeping the flow steady and channels balanced.

Sinks And Sources

Global flows in the Lightning Network take more or less established patterns. This results in a certain prevailing direction of the local flow in each of your channels. A peer that tends to drain your local balance is a sink, and the one that insists to keep that balance high is a source. If a peer is a source to you, you are a sink to them, and vice versa. Only sink channels generate decent routing fees, but need to be rebalanced to keep working. Sources require much cheaper fee rates to create inbound capacity, but when that fills up they earn “assisted revenue” for other channels.

The prevailing method of rebalancing is “circular”: spending high local balance in source channels to replenish the drained sinks, for a fee. As discussed in my previous article, this method preys on smaller less experienced nodes that are content to earn lower PPM. It is unfair, but thoroughly capitalistic. When left unchecked, routing other nodes’ rebalancing flows will unbalance your channels, necessitating a rebalance, and so on. I’d say 90% of LN overall traffic are rebalancing flows of some sort. Your goal is to live with that and adapt.

The trick is to use those active rebalances to passively rebalance your own channels. This does not seem hard, because the initiators share the same goal — bring all channel balances closer to 50/50. But you are rarely a direct peer, so the flows you receive seldom fit. You would want them to take excessive balance from your source channels and add to your low level sinks. The rebalancing node has budgeted some max PPM, and you want to squeeze just under it. If successful, such passive rebalancing improves your node’s capital position without costing a sat.

The intuition is straightforward: set outbound fees low on the channels with excessive local balance, except freshly opened ones. For a new channel, start with a high fee, like 2000 PPM, and decrease it gradually every day to see when it starts flowing, then fix it. For sink channels with low local balance set the outbound fee prohibitively high, so that the route finding algorithm knows not to test them. Failed HTLCs temporarily blacklist a channel from route finding, which is not a desirable thing. If the “Insufficient Balance” failure occurs, it is best to bump the fee even higher.

It is worth mentioning that preventing failed HTLCs by reducing Max HTLC parameter is not a good idea. This will not stop the liquidity from draining, just make the chunks smaller. Signaling with a higher outbound fee does not have this drawback and does not give away information about your actual balance changes. Instead, you gain information at what highest rate your channel can potentially flow. So that when the local balance is replenished, you will continue offering it at the highest rate.

For the same reason, you don’t want to increase the fees too often. They take a few hours to propagate via gossip network, so until they do HTLCs will arrive with the old fee rate and cause “Fee Insufficient” failures.

However, you want to react quickly when your low promotional rate attracted the flows, and the channel no longer has an excessive balance. You don’t want to sell your whole liquidity at that low rate. Even if no one has been interested for days, a new node can open a channel any minute (usually while you are asleep) to take advantage if low rates. This is called “liquidity sniping”.

A Three-Tiered Approach

The channel should stop offering a promotional rate the moment the balance returns to normal, and when the balance gets too low it must signal that with a very high fee. What is “high” and what is “low” balance, as a percentage of capacity, is subjective. For a 20m channel 5% is 1m sats. It is low if a channel sees a lot of jumbo sized HTLCs, so your fee management software should be able to catch these failures and move the threshold higher.

For a 1m channel, a 50% can already be considered low balance. So the rules must be flexible to adapt to every channel size. The high liquidity threshold can be set at 70–90%. On one hand, you don’t want to give away lots of cheap liquidity. On the other, liquidity trapped in source channels’ balance is cheap for a reason: that capacity is idle.

The three tiers of liquidity command three levels of outbound fee rates, with a step function between them. It is tempting to instead increase the fees gradually and proportionally as the channel gets depleted, but this would result in gossip spamming and higher failed HTLCs statistics. You are better off routing the next HTLC with the same “normal” rate than failing it trying to earn extra x%. Therefore, it is crucial to figure out that standard fee rate for each of your channels. Studying the channel’s historical forwards realized PPM will help with that decision. The high liquidity rate will be below it, and the low liquidity rate way above.

Inbound Fee Discount

Until recently, setting a low outbound fee rate was all you could do. And a typical result was certain source channels routing to other source channels cheaply. A sink channel with low liquidity did not budge. One solution was an active circular rebalancing, spending lots of time and money, budgeting higher and higher Max PPM on each attempt. Another novel approach is to use Liquid PeerSwap if the peer has it installed and you have L-BTC balance. Finally, the newest trick you can try today with LND 0.18 is to set negative inbound fee rate for that sink channel.

It does not mean you will pay a fee for inbound traffic, this is technically impossible. Instead, you will offer that peer a discount on any outbound fee used in a forward. So, if the rate is 50 PPM on some channel and the inbound discount is -50 on another, the resulting cost will be 0 to route in that direction. If the discount is larger than the fee, the cost will stay at 0.

The way to properly use it is to have relatively high outbound fee rates for all channels, and give serious discounts to the most stubborn sinks. A fee management algorithm should be able to set such inbound discounts automatically when the liquidity dips below a pre-set threshold. Of course, currently this will only work for the routes where planning is done by another LND 0.18 node.

Tools Of The Trade

I don’t know much about other node implementations, but the two most popular fee management instruments for LND are LNDg and charge-lnd. LNDg is a great web UI suite for monitoring and auditing your node. Its fee managing algorithm has the concept of low, medium and high liquidity tiers, and it can bump the rate on failed HTLCs. But all the parameters are general for all channels at once, with no customization. There is no inbound fees in the current release v1.8.0 and the web design is not mobile friendly.

Charge-lnd allows more flexible policy definitions, but it lacks the UI. One needs command line skills to install and configure it, with all the parameters managed via a text file. It is hard to master and difficult to adjust on the fly. Both LNDg’s Auto Fees and charge-lnd are Python scripts that run periodically on timer. They are unable to react quickly to up-to-the minute changes in channel balances to prevent liquidity sniping.

Fortunately, automated channel fee management is now part of PeerSwap Web UI - a high performance software written in Go and compatible with both LND and CLN. It combines flexibility of charge-lnd with the convenience of LNDg, fully implementing the strategy discussed above. If the novel and cheap method of channel rebalancing with Liquid was not a sufficient incentive for you to try this new tool, now I give you one more!