As Free as the Breez: Building Lightning to Make Bitcoin Mobile
By now, Clarke’s third and most famous law has firmly entered the tech vernacular:
Any sufficiently advanced technology is indistinguishable from magic.
But that’s not the end of the story. First comes that thrilling moment when we marvel at the new technology, when we remember how its functions used to be impossible and are now effortless, and we appreciate the magic. That honeymoon phase might last for a few months. Then we come to rely on the magic, forget what life used to be like without it, and get annoyed when it doesn’t work perfectly. Alas, the fate of all technological magic is to become a utility.
Smartphones have certainly followed this trajectory. Such devices used to be part of the furniture in our utopian visions, appearing in primitive form as the Hitchhiker’s Guide to the Galaxy, the tricorder from Star Trek, and some of the iconic future tech from 2001: A Space Odyssey. After having lived with “real” smartphones for 13 years now, they’ve become part of the furniture of our mundane reality. Not being able to reach any function within three taps? What is this? 2006?
Don’t get me wrong. This is a triumph. It’s something to be proud of … as well as a burden. Any new mobile tech must match the brilliant, magical UX we’ve come to expect on our smartphones, or nobody will use it. Progress is a one-way street.
From an engineering perspective, users’ expectations are a constraint. Any solution must work within its constraints, or it’s not a solution and we go back to the drawing board. The good news is that we’re engineers, and deep down in our cold, binary hearts, we love constraints. They push us to optimize, to chase after ever more efficiency and stability. Building a Lightning client to run on a quantum computer cooled nearly to absolute zero is for amateurs. If you want a real challenge, try building one that will work on any number of off-the-shelf devices running different versions of iOS and Android. Oh, and keep it standard, secure, private, fast, non-custodial, and easy to use.
Since building a mobile Lightning client is such a challenge, and since we’re occasionally asked how we make Breez so functional and simple, we thought you might like a peek under the hood. Here are some of the obstacles to operating a mobile Lightning client like ours and some of the particular solutions we’ve deployed to overcome them.
Integrating & optimizing Neutrino, whatever the CPU says
While there are significant differences, and in spite of the zealotry displayed in their respective fan communities, Android and iOS have something in common: they are the hall monitors of our mobile devices. No app is allowed to do anything without their permission. “You just used network connectivity, so you’ll have to wait until next period.” “Where’s your hall pass to access CPU cycles?” These guys are the life of the party.
The value of this supervision is to make sure that the combination of Candy Crush and TikTok doesn’t melt a phone in a user’s hand. Fair enough. But it also presents an almighty challenge when it comes to running a Lightning node on a mobile device. Lightning nodes were originally designed without CPUs or networking constraints in mind. For example, a Lightning node needs to constantly sync with the blockchain in order to validate channel states. Nodes also need to keep the network graph current and constantly broadcast channel information to peers.
The first obstacle is how to access the requisite CPU time while running as a background task, and it’s an obstacle because Breez uses Neutrino. Any non-custodial bitcoin service needs to stay current with the public ledger, and Neutrino is simply the most private means of doing so. Breez was also the first Lightning client to integrate Neutrino. Please hold your applause.
The challenge was and remains that Neutrino requires the latest blockchain filters to be downloaded and updated almost constantly, which takes connectivity and CPU cycles. Breez schedules a regular background task to fetch new chain data, but iOS and Android treat background tasks like extended family: some are always welcome and others are politely ignored, though it’s hard to say which or why. CPU access is determined by a mystifying combination of battery status, power supply, other concurrently running apps, and proprietary algorithms. Expecting users to wait until the CPU gets around to processing Neutrino’s requests before they can use the Breez client is unacceptable. Users don’t wait and shouldn’t have to.
To help our Lightning client function optimally in the world of CPU constraints and user expectations, we have invented a few helpful processes.
- We’ve retooled Neutrino to allow it to work independently of lnd. As a result, we reduce the amount of code and data competing for that precious CPU.
- We’ve optimized lnd to allow outgoing payments even when the client is not synced, which is important because sending funds is the standard use case for most of our users (i.e. those not using our killer point-of-sale mode). Sending funds while out of sync does not compromise their security, although receiving funds while out of sync might.
- We sync Neutrino only from the last available mined checkpoint. It’s like catching up with a friend’s news since your last meeting, not rehashing your entire mutual history at every encounter. This results in less data to move and crunch, making Breez faster and lighter.
- We save the requisite Neutrino filters to disk, which allows them to be quickly referenced locally instead of having to download them anew in every instance. Of course, they are also automatically deleted when no longer needed.
- We’ve implemented a wetware failsafe. Should Neutrino fail to access the CPU for three days — for whatever reason — the app notifies the user that Breez is 72 hours out of sync, and it needs some attention and action on the user’s part. This probably just means that the user needs to actually open the app, prioritizing its demands on the CPU’s time.
Routing, graphing, pruning & zombies
In order for a node to route a payment through the network, it requires a map: the network graph. Although there are promising proposals to liberate routing from the network graph, source routing remains the only way to route payments privately (such that only the user knows the payment destination).
In the current implementation of lnd, nodes broadcast their channels’ status every 24 hours. That’s fine for servers, which have enough uptime to ensure that they will receive most of these updates. Mobile nodes, however, spend most of their time offline, so they are likely to miss many broadcasts. Eventually, these out-of-date channels are considered “zombies”. Just like Hollywood’s zombies, it’s much easier to become one than to recover from zombiehood.
The problem is that payments can’t be routed over zombie channels, so Lightning “prunes” them (“pruning zombies” — what a metaphor, I know, but we’re engineers, not poets). However, the default mode of identifying zombies is biased against mobile nodes, potentially infecting many innocent, upstanding channels with the zombie virus and cutting them off from the network.
Our solution was to change how we treat these channels. Instead of assuming that they’re zombies and pruning them right away, we assume they’re valid and modify what data the mobile nodes query from the full nodes. These small tweaks mean that, in some cases, thousands of zombies were cured, resurrected, and returned to the graph of active, valid channels.
We also had to consider how to keep the graphs on our users’ mobile nodes up to date. Their downtime is the root of this challenge, after all. To keep them current, the app will initiate a fast graph-sync procedure in case the graphs on the user’s light client fall a few days behind. As a result, the app is ready to broadcast payments using up-to-date graph information within seconds of opening instead of the minutes it usually takes for graph updates to be received over the Lightning channels.
As the network scales, source routing based on a complete graph will become increasingly impractical. Just imagine the labyrinth of channels available if even 10% of all people start to use Lightning regularly. Therefore, it’s important that we keep our eyes on alternative routing algorithms, like ant routing and trampoline payments, as well as their effects on privacy.
Syncing the wetware
It’s not only the CPU’s attention we need. Sometimes we need access to the users’ attention as well. For example, the payer and payee of a Lightning invoice need to have their clients open at the same time in order for the payment to go through. As a result, Lightning payments are more like telephone calls, where communication only works if both sides engage simultaneously, than like messaging, which allow asynchronous conversations.
The minimum requirement, therefore, for a functional non-custodial app is some means to get both parties’ clients online at the same time. To achieve simultaneity, we developed Connect To Pay. It works great. Connect To Pay basically lets the payer send the payee a message that the payment is coming, informs the payer once the payee has opened their client, and then lets her input and transfer the desired sats. Watch the video. It’s about as intuitive and difficult as putting on socks.
Modelling the UX on the telephone call is good as far as it goes, and our feedback indicates that normies worldwide love it. But remember the good ol’ days? Back when we used to sit in airplanes and cinemas, which are times and places when using your phone is not possible or just rude? And having to wake up just to accept a payment seems suboptimal.
That’s why we’ve invented Lightning Rod. Once it goes live, Lightning Rod will allow the payer to initiate a payment without requiring the payee’s immediate attention. The payee will be able to receive the payment at her convenience. The secret is to use hodl invoices to give the payee more time to accept the payment before the HTLC runs out and the transaction expires. Just like a text message or an email, the payment will wait safely and validly until the payee is ready to deal with it.
Why all this trouble?
Getting things right is demanding. There are few real shortcuts in life, which is an old idea. One parable in the Talmud that was only recorded about 1800 years ago expresses it like this:
What is the incident with a young boy? One time I was walking along the path, and I saw a young boy sitting at the crossroads. And I said to him: On which path shall we walk in order to get to the city? He said to me: This path is short and long, and that path is long and short. I walked along the path that was short and long. When I approached the city I found that gardens and orchards surrounded it, and I did not know the trails leading through them to the city.
I went back and met the young boy again and said to him: My son, didn’t you tell me that this way is short? He said to me: And didn’t I tell you that it is also long?
Devising, coding, testing, debugging, integrating, and explaining all of these innovations is the long (short) path. There is a shorter (longer) one: an intern with spreadsheet. Yes, we could simply call ourselves a custodial service, take possession of our users’ funds and pay an intern as little as possible to shift balances around on a spreadsheet. Even at 12 Red Bulls per day, it would be cheaper and easier than having competent, passionate people coding all this. Many users would not notice the difference in the UX.
But that easier path doesn’t lead to our destination. We’re aiming for a bitcoin economy. Bitcoin needs to remain not just decentralized, borderless, neutral, immutable, and publicly verifiable, it also needs to be private. We don’t assume that you’re okay with us knowing about your transaction history. And we don’t really want to know. But we do know that privacy, which we love, hinders censorship, which we deplore.
That’s why Lightning immediately inspired us. We recognized its potential: it’s 100% bitcoin, but it provides the speed and economy bitcoin needs to go mobile. To go global. To go universal. The problem was just that Lightning wasn’t yet finished. It was only a framework, albeit a brilliant one. We knew that many people would have to invest countless hours to add functionality and bring the UX to where it needs to be, and all without sacrificing the principles and convictions that make bitcoin worthwhile.
You see, some apps just “implement” Lightning or not even that — they just take custody and do who knows what while calling it “Lightning”. Not us. We in the Lightning community are building Lightning, identifying problems, concocting solutions, and giving that knowledge away to our users and even to our competitors. Because Lightning is about making bitcoin mobile, not just giving people another reason to play with their phones.
