What We Learned from Sending over 1,000 Lightning Bitcoin Payments
Back in Feb 2017, a co-worker and I set up TokenSoft’s first Lightning node during a late night hackathon while on a skiing trip in Tahoe. We didn’t quite understand how it worked or what it could be used for in the future, but we both thought it was a really cool technology. Looking back, it was truly #Reckless to put money into livenet as we ran into many issues while testing and playing with it.
Our setup consisted of a Bitcoin Core node and CLightning. There are many tutorials out there to get it set up, and even hardware specific devices that come preconfigured (shout out to Casa). For our specific configuration, we manually installed both the node and the lightning services on an Ubuntu server in AWS and let it sync up.
Once it was synced up and we were up and running, there wasn’t much you could do with it, but we connected to Blockstream’s node and bought stickers from the store. Also, we set up our own store, where anyone can buy a TokenSoft “Ride the Lightning” mug.
Our initial thought was to create the biggest node on the network. Back when there were only 100 or so nodes on the network, this was very feasible, but today it would require much more capital committed and a very beefy server.
We ended up building out our own version of AutoPilot for CLightning. LightningD, one of the other implementations of a Lightning node, already had this functionality built-in, but on CLightning you had to handle it yourself. We ended up building a very simple implementation that scanned the entire list of nodes on the network and created a channel starting with the nodes that had the highest number of outgoing connections and working our way down.
At one point, we had the most connected node on the network with around 200 outgoing channels and around 5% of the network’s entire BTC balance locked up in them.
By pushing the limits of the new software, we definitely uncovered a bunch of edge cases (e.g. these issues). Many of the channels we tried to create ended up being invalid BTC transactions and the Lightning node would broadcast them, the bitcoin node wouldn’t relay them, but the Lightning node would sit around and wait for them to get confirmed, ignoring the error. This would cause the lightning node to be out of sync with the real state of the channels and essentially have a corrupted db.
When pushing the limits on the number of nodes we were connected to, crashes during startup became an issue. Corrupted channel states plus nodes coming up and down seemed like a recipe for creating edge cases in the database values when interpreting the current state of a channel. Many kudos though to the Blockstream team and other contributors for helping us diagnose these issues.
Overall, we never ended up losing funds, though at one point we did think we had lost around 0.2 BTC due to corrupted channels. We had to manually kill and recover locked up funds to get back to clean state. Eventually, the database got so corrupted we pulled all the funds out and started from scratch with a clean database. This did mean we closed all of our open channels and connections to other nodes, but at the same time, we gained peace of mind that we wouldn’t lose any funds.
The Satoshi Test
Once we got up and running with our fresh node, we decided to have some fun and use the node for “marketing”. We connected to the Satoshis.place node and pasted our company logo front and center for all users to see when the page loaded.
Immediately someone “defaced” our logo and showed us this was definitely a permission-less environment. Not to be outdone, we decided to show the script kiddies who was boss, and built out a tool that would ensure our pristine logo remained on the site.
The code we built converted our logo from a PNG image to a matrix of pixels with RGB colors. We then requested an invoice from satoshis.place to paint all these pixels and submitted the lightning payment to get it on the site. Next, we hooked into the event subscription on the site to get notified any time someone writes pixels, and we do a diff of the new pixels versus what we want painted. If anyone ever writes over our logo, we immediately trigger an invoice payment to replace only the pixels we need to get our logo back to a pristine state.
Over the last few months, it’s been pretty funny watching random people deface our logo, only to have it snap back to a clean logo seconds later. We’ve had many people create a “war” with going back and forth writing over our logo. We’ve seen people calling our logo a “scam” for whatever reason. And we’ve also seen people have fun extending our logo to include additional fun stuff. To this day though, anyone who navigates to the site will be presented with a TokenSoft logo, front and center.
Last month, our current lightning node hit over 1,000 payments to Satoshis.place. It has been surprisingly stable after getting to the latest version of CLightning. We do have a direct channel to the Satoshis.place node, so we are not requiring the routing logic to send payments through the network, which has been a constant weak point and critique of the technology.
The usability aspect of getting set up and connecting channels is still a challenge, though many teams are working on consumer-facing wallets trying to address this. The mental friction and time period for initial channel creation may be a barrier of adoption for the consumer side and it may turn out to be more of an enterprise tool because of this.
Ignoring some of the initial friction and issues we faced, Lightning has proven its applicability for frequent and fast micropayments. For use cases like balancing accounts across exchanges for arbitrage, paying by the minute for service use (like internet bandwidth), and frequent payouts for non-custodial account balances (like arcades or casinos), it definitely seems like it has a future.
We look forward to seeing the technology evolve. Ping us on Twitter and let us know if you see any funny business with our logo.