Adventures in Running a Bitcoin Lightning Node — Part 2: Growing up…

Photo by Wes Carpani on Unsplash

A lot has happened since the first article Adventures in Running a Bitcoin Lightning Node — Part 1: The Awakening… of this “potential” series:

1. The world moved on…who knows, hopefully to the better!
2. Bitcoin did die…again!
3. My lightning node routed along…well sooooommmmetimes!

But: I learned a ton! And as much as I want, sharing all of it is hard to recapture and might be too detailed. So, here is the ultra-short summary of some insights you hopefully do not already know:

1. Finding good channel peers: I tried to find good peers multiple ways. Lightning Network+ and Rings Of Fire communities were good to find peers to connect to and at the same time get inbound liquidity. However, not all channels from those connections really worked out, roughly 80% of them are just not flowing very good so far. But the other 20% became good channel peers.
   I also played around with my own strategy to connect with the top 200 nodes ranked by Terminal where I share the least number of peers in common in order to be part of unique paths on the network. However, I guess when your node centrality is not high enough, unique paths do not really attract a lot of routings by itself. Valuable channels did not result from this strategy so far.
   What worked best so far is simply to open channels larger than 3M sats to peers very highly ranked by Terminal (top 50) whose fee policy fit to your fee policy.
   Recently, I took a closer look into Magma of Amboss. An additional option to explore!
2. Automate charge-lnd and rebalance-lnd: Initially I applied proportional strategy of charge-lnd that sets the fee according to the balance of a channel to all channels (i.e. low fee when channel liquidity is local, high fee when channel liquidity is remote). Rebalance-lnd was applied in the beginning only manually to observe which channel are rebalancable at what fees and amount. After a couple of weeks, I got insights in at what fee rate channels flow and which of them are rebalancable at what rate. Successively, I applied a static strategy of charge-lnd for predictable channels with a fixed forwarding fee. Additionally, in order to distract other peers to use a depleted channel (i.e. almost all liquidity sits remote) for routing and to avoid failing forwarding attempts, one can set max_htlc_msat to a value lower than the available local balance (be careful: by observing publicly visible changes of this value one can guess how much volume is routed, and therefore this can be a potential privacy issue) and/or can set base_fee_msat/fee_ppm to very large values.
3. Tor and its connectivity issues: Tor is great if you do not have a static IP and the knowledge to apply proper OpSec to secure your node behind a public IP. However, stability of Tor is not always a given and can have negative impact on your node’s connectivity and routing behavior. During June 2022 an ongoing Tor Network DDoS attack lead to observable less routing activity, more frequent channel disablement and unfortunately also to some force-closing of channels of my lightning node. To circumvent the Tor issues, I set up my node for hybrid mode (Clearnet and Tor) connectivity. This noticeably stabilized connectivity! However, setting up a node in hybrid mode requires you to be tech-savvy enough. Of you don’t feel comfortable enough to handle that, I can recommend investigating recent services like TunnelSats (be aware of the privacy implications!).

Ok, so far on the ultra-short insights, lets get to some real gem…

Wind-up source-sink strategy

To my knowledge, there is no established well-structured classification of strategies to operate a lightning routing node. A transcript of a talk of Alex Bosworth from March 2021 comes closest of being the first form of classification. In his talk, Alex defined seven types of routing nodes: Ping-pong; Liquidity Battery; Inbound Sourcing; Liquidity Trader; Last Mile; Swap; Data Broker. What I describe in the following as so-called wind-up source-sink strategy is a specific shaping of a liquidity trader routing node strategy:

Wind-up source-sink strategy of a lightning routing node

The goal to achieve profitable forwarding earnings of the wind-up source-sink strategy of a lightning routing node is as follows:

A. Establish an always-available forwarding opportunity of payments for a source peer to a sink peer for a reliable (e.g., static) and profitable routing fee of x ppm,

B. by automatically circular rebalance from source to sink channel for a reliably and significantly lower rebalance fee of y ppm (y << x).

Conditions that should be given to breed a profitable wind-up source-sink strategy:

1. Long-living source/sink channels to very reactive, always-available & strongly pushing source/pulling sink node with predictable fee policies. Long-living -> to save on open/closing costs; always-available & predictable: to establish reliability; very reactive & strongly pushing/pulling -> to repeat the pattern as fast as possible.
2. The sink node immediately pulls new available outbound liquidity on sink channel (high throughput, usually never sends any sats back). To repeat the pattern as fast as possible.
3. Forwarding (=rebalance) fees of source/sink nodes are highly predictable. Reliability/Profitability through predictable rebalance fees.
4. Source (and sink) node are well-connected. To have a multitude of potential rebalance routes.
5. Source/Sink nodes do not close channel when strongly imbalanced. To repeat the pattern reliably and profitably.

For someone who is familiar with the rebalancing principle this sounds like a rather obvious strategy. What surprised me was to observe …

1. that such source-sink pairs exist, where the rebalancing costs are proportionally much lower — up to 60% — than the earnings on forwarding and
2. that such source-sink pairs exist in its most reduced and simplest form — meaning that the source and the sink channel are used for the rebalance operation without using additional other channels of your node. Therefore, this strategy can be applied two exactly two channels without interfering the behavior of other channels.

How to find a source-sink pair for the wind-up source-sink strategy?

So, where, and how to find such a pattern? Here is some general advice resulting from my experience:

1. Observe your channels to find a sink channel:
   a. Is there a sink channel that regularly pulls your outbound liquidity but never sends back?
   b. When you rebalance to it, is your newly gained outbound liquidity at best immediately pulled again?
   c. Was your forwarding fee for the pulls the same and constant? How do the forwarding fees of your sink channel change for various balance states of the sink channel? Do they change predictably? Can you observe an upper bound for the rebalance fee that you can set and still rebalance immediately or at least very fast and reliably?
   d. Is the upper bound for rebalance fee significantly lower than your forwarding fee for that channel?
   Than this could be a candidate for a sink channel for your wind-up source-sink strategy!
2. Observe the forwards and rebalance routes for the sink candidate:
   a. Do you have forwardings to your sink candidate that always come from the same source channel?
   b. Do you have rebalance routes that start very often from the same channel?
   c. Do you identify an incoming forward source channel that also acts as a rebalance-out channel?
   Than this could be a candidate for a source channel for your wind-up source-sink strategy!
3. Validate your source and sink candidates with automated rebalance scripts:
   a. Identify a good lower bound for the outbound liquidity on the sink channel that you are able to constantly rebalance and that is enough that the demand of the sink can constantly be served.
   b. Identify a good lower bound for the inbound liquidity on the source channel that is not used by other forwards and that is enough that the demand of the sink can constantly be served.
   c. Beef-up and apply your rebalance scripts (I use extended bash scripts based on rebalance-lnd) such that they always start rebalancing immediately from source channel to sink channel whenever your lower bounds you have defined for source-sink channels are threatened to be breached.
   d. Can you constantly provide source-sink flow within your bounds with your automatic rebalance scripts? Does the source-sink flow continue over a longer time period (e.g., days, weeks) without forwarding fee reduction? Are your rebalancing routes and is your investment in rebalancing stable?
   If all answers are Yes, BINGO!

How to optimize and scale your wind-up source-sink strategy?

Consider the following questions as next steps:

• Can you optimize the source-sink flow such that no forwarding interrupts happen (because channels temporarily depleted)?
• Can you further improve by constantly adapting max_htlc (in cases where the capacity or balance of your channels are hindering)
• Can you scale the demand of the sink?
• Reopen larger source or sink channel?
• Reopen more channels to nodes of a sink super-node (if it is a compound of lightning nodes)?
• Can you scale the supply from the source node?
• Can you scale your rebalancing capacities? I was a bit hesitant to apply rebalancing too aggressively. But at the moment, my node sends out rebalance attempts constantly without any issue so far. So don’t be too shy!

But be cautiously:

• There is no guarantee that the source-sink flow and profitable rebalancing possibilities sustain!
• Aggressively rebalancing can potentially have negative effects on your node (e.g., lot of past channel states, CPU cycles, influence on mission control) or on the likes of your routing peers (peer closes channel because annoyed by many incoming HTLCs).
• Finding a pure source-sink pair where no other peers interfere is not so easy because they are probably rare.
• If the lightning network matures further, the profit from such imbalances in the network probably is decreasing. I would not invest heavily on that strategy solely. Also apply other profitable strategies with other channels.

Otherwise, slurp the juice!

Results

• I applied the wind-up source-sink strategy now in the last two months where possible and it increased my earnings through forwarding to sinks very significantly!
• Depending on the profit range y-x ppm and how fast the wind-up pattern can be repeated over time, for me this yields an APR of 0.5% up to 1% for the sink channels. But having the right source-sink channels, much greater APR should be possible!
• To be precise, my APR calculation follows the following definitions:

The APR formula shows what is relevant:

1. The larger the difference between forwarding fee y and rebalance fee x is, the larger the APR.
2. The smaller f is (i.e., how fast the pattern can be repeated), the larger the APR.
3. Channel capacity must not be necessarily large, just large enough to keep the wind-up source-sink pattern going without liquidity shortages!

Already right, so far for this time!

Hope you found this article insightful! Ping me if you have any questions! Stay tuned for more…