Erlay: An awesome performance improvement to the Bitcoin network
I’m writing this non-technical explainer because I believe Erlay is one of the most important performance improvements to Bitcoin yet.
Erlay will help the Bitcoin network to remain decentralised, by reducing unnecessary communication between the participants in the network.
Thanks to Gleb Naumenko, Pieter Wuille, Gregory Maxwell, Sasha Fedorova and Ivan Beschastnikh for proposing this awesome improvement.
I first read about it in the (highly recommended) Bitcoin Optech Newsletter (#49). Special thanks to the authors for making it so accessible to a somewhat technical audience. I hope to make it even simpler for others below.
How broadcasting a transaction works today
This is Bob.
Bob wants to send a bitcoin transaction, the reason why isn’t relevant for this explainer, but you can make an educated guess.
He obviously wants his transaction to be included in the Blockchain as soon as possible. To achieve this, it helps if many of the participants in the Bitcoin network know that Bob’s wallet has broadcast his transaction. This will increase the chance of his transaction being included in the next block of transactions that is created, which happens every 10 minutes on average.
As Bob is more interested in doing other things with his money, he does not have his own node in the Bitcoin network. Instead, he trusts Alice’s node to broadcast the transaction it receives from his wallet.
Alice’s node first checks Bob’s transaction, to see if his wallet signed it.
It also checks whether Bob has spent this bitcoin before, by looking through its records of currently unspent transactions, known as “UTXOs”.
The “What are UTXOs” explainer
A bitcoin transaction has both an input (your bitcoins on an address controlled by you) and an output (the receiving address).
Every transaction output, can in turn become an input for the next transaction.
To make sure you’re not spending the same bitcoins twice, the node could check its entire history of hundreds of millions of transactions to find if the same transaction input has been used before.
As you can imagine, that is a lot of work to do every time. To be more efficient, the node will go through the entire history once, to create a list of all of the transaction outputs that have not been used as transaction inputs yet.
This is known as the list of Unspent Transaction Outputs, UTXO for short.
Today in Bitcoin there are around 56 million UTXOs, which take up around 3GB of storage space. The entire Bitcoin Blockchain on the other hand is about 220GB, and contains over 422 million transactions.
Each time a transaction is sent, a node removes old UTXOs that were spent, and adds new ones, to keep the list up to date.
After everything checks out, Alice’s node tells the 8 nodes it is connected to about Bob’s transaction.
Bob may be trusting Alice’s node blindly, but its peers don’t. They will perform the same checks Alice’s node did on Bob’s transaction.
As you can imagine, many nodes will hear about Bob’s transaction multiple times after that, as they are all connected in different ways and don’t know who receives which information.
A lot more messages are being sent than necessary. While this gives Bitcoin users a high degree of certainty that all nodes in the world will know about their transactions, it puts the burden of this luxury on the nodes.
These nodes send and receive far more data than necessary. To be precise, the research paper shows that 44% of all the traffic between nodes are these kinds of unnecessary messages!
This adds a lot of pressure to nodes, especially those that may not have amazing Internet connections, or whose owners can’t just keep paying more and more for their Internet subscription as Bitcoin is used more.
This problem is addressed by the proposed Erlay protocol.
How broadcasting a transaction works with Erlay
While you were busy learning about UTXOs and the extra traffic for nodes, Bob has learned how to set up his own Bitcoin node from a tutorial.
Bob has made Alice one of his peers, and broadcasts his transaction to her and 7 of his other peers. After verifying his transaction, Bob’s peers also tell 8 of their peers.
Once this transaction has gone across the network, not every node has received Bob’s transaction. If you were not included in someone’s group of 8 peers, you may have missed out.
Not to worry, a node doesn’t have to hope it will hear about Bob’s transaction. It will instead periodically request a sketch of all the transactions that all of its peers have received.
- Each peer sends a sketch that represents all the transactions it received, which takes up less data than sending every single transaction in full.
- The node will then produce its own sketch of the transactions it has received, and compare this sketch to the ones it received, like a game of ‘find the difference’.
- The node can then request the exact transactions that are missing from its own sketch, from its peers.
A good metaphor to understand the difference between the sketch and the missing transaction is a panorama view of a landscape (all of the transactions) or a detailed close up of an individual flower or rock in that landscape (one transaction).
Advantages and disadvantages
There is one drawback in comparing these sketches to find the differences. It takes slightly longer (2.6 seconds) for a transaction to be known by all of the nodes in the global Bitcoin network.
Since Bitcoin gets a new block every 10 minutes on average, this slowdown seems well worth it to alleviate much of the heavy work that nodes do.
If in turn, nodes have to do less work, then it becomes easier for people to run their own node, and the Bitcoin network can remain decentralised.
We would not want a situation in which a few big data centers are the only ones capable of running a full node, as all of us would have to trust them for our transactions. That puts us back to a world of centrally controlled financial institutions, while Bitcoin was invented as an alternative to that.
The “But what if” box
You can try to think of ways how this Erlay protocol could fail, but so have the authors of the paper!
In there, they describe the optimal parameters to compare these sketches and what steps a node can fall back on in case it doesn’t manage to find the differences between the sketches.
The authors also tested the performance of this protocol on a simulated network of 60.000 nodes and a live network of 100 nodes spread across the world.
If nodes would increase their number of peers from 8 to 32, they would normally have a 300% increase in traffic. With the Erlay Protocol, this is just 32%.
So when can we see Erlay being included in Bitcoin software?
Right now, the research paper is open for feedback.
Edit: A Bitcoin Improvement Proposal draft version for Erlay has now been published as well! (26 September 2019)
Edit 2: An implementation is now up for review as well! (4 March 2020)
If no objections are raised by other researchers, testers or developers, Gleb Naumenko plans to write a Bitcoin Improvement Proposal, and start the work on getting Erlay included in the most popular version of Bitcoin node software: Bitcoin Core.
Erlay would only require 584 lines of code. It doesn’t require any fundamental rule changes in Bitcoin that would make the new version of the software incompatible with older versions, so getting it activated will be much easier than some of the other changes to Bitcoin that have been made.
I hope you enjoyed this explainer, it was a bit of experimentation for me. If you liked it or have suggestions for other Bitcoin-related topics, let me know!
