Mempool size limiting

An FAQ for Bitcoin XT 0.11B

This article describes a new feature in the XT 0.11B release, called mempool size limiting. 0.11B is due to be released tomorrow, but you can already get the downloads we plan to release here.

This new feature exists to handle the transaction flooding denial-of-service attacks, such as the one scheduled for 11am CET on Thursday by “James Wilson” of CoinWallet.eu (both name and site are certainly fake).

Q: What is mempool size limiting?

The mempool is the network’s holding area for transactions that have been made but not put into a block yet. Every Bitcoin node on the network holds unconfirmed transactions in RAM and removes them once they are confirmed. From then on they’re held only on disk.

Bitcoin Core and releases of Bitcoin XT prior to 0.11B have no limit on the number of unconfirmed transactions they will try to hold. If a transaction is valid it will stay in RAM forever until it’s mined. This means DoS attackers can flood the network with transactions until nodes start to run out of physical memory and fall over. I wrote about what can happen then in my article, Crash Landing.

Q: What is new in XT 0.11B

In XT 0.11B the mempool size has a configurable limit. When the mempool is too big and a new transaction arrives:

1. If the new transaction is low fee/low priority, it is dropped. “Low fee” means it doesn’t pay high enough fees to get in within the next two blocks according to the fee estimator.
2. Otherwise space is cleared for it by selecting a random transaction and any that depend on it, and then removing them.

Note that the transaction selected is different for each node. Therefore, when the mempool is full and a new transaction is broadcast, each node will forget a different set of transactions.

Q: How can I control it?

The feature is controllable with a new flag:

./bitcoind -maxmempooltx=50000

This will set the limit to 50,000 transactions at once. As transactions can vary in size, the total amount of memory used may be higher: an average transaction is 500 bytes but during DoS attacks the average may go up.

If left unset, the default is enough transactions to fill 25 blocks of whatever the current max block size is.

Q: What if something goes wrong?

The feature was developed quickly in order to be ready in time for the scheduled DoS attacks. The code has been reviewed and tested by several experienced Bitcoin developers, but if anything goes wrong, the feature can be disabled completely by setting the -maxmempooltx flag to zero.

Q: Why do this rather than sort-by-fee?

For two reasons.

In the ideal world you always have limitless time to debate, test, review, experiment and so on. But engineering is about tradeoffs, so sometimes you have to go with something simple-but-quick over something complex-and-slow.

The code to do random selection is dramatically simpler than the code to sort the mempool by fee/priority and cull. Because it’s much simpler it could be written, peer reviewed, tested and released much faster, which is important because we are currently under attack.

The second reason is a bit more controversial. I am not convinced that sorting by fee is actually a better solution at all.

Imagine for a moment that block finding happens precisely every ten minutes. We can see that the memory pool should ideally never have more than one block’s worth of transactions in it because if it did, that would imply payments were being made faster than Bitcoin can clear them. Actually blocks are found every 10 minutes on average so sometimes we can go an hour or even two hours with no block being found just through bad luck. So the memory pool should be able to handle some backlog that we’ll catch up with later when our luck turns good again.

But a situation where the memory pool is permanently and massively backlogged should never happen ….. unless there is a DoS attack.

In a DoS attack, most of the mempool is full of attack transactions that don’t represent any real economic activity. As blocks are on average about half full today, with the XT default settings during a DoS attack of average-sized transactions we can see that the mempool only has around 2% legitimate transactions and 98% of them are spam.

Therefore just by picking a transaction at random, we are very likely to delete one that nobody will miss. As a bonus, DoS attacks often create huge chains of transactions that depend on each other. If we get lucky and pick a spam transaction that’s towards the bottom of such a chain, we’ll erase lots of guaranteed-to-be-spam transactions simultaneously, freeing up lots of space in the mempool for legit traffic. If we do pick a legit transaction, other nodes will have probably picked differently anyway so at worst the user will just have to wait longer for confirmation, which is the best we can do.

Bitcoin Core is pursuing a different approach, where the memory pool is ordered by fee and then if a higher fee transaction comes in, it jumps the queue and whoever is at the end falls off. This makes sense given their small-blocks view of the world. They want the Bitcoin network to be permanently full because they want to establish a fee market where Bitcoin users compete against each other for the, um, privilege of making payments with Bitcoin. Their logic is that if transactions were free, demand would be infinite. Therefore this more complex approach is required.

It’s hard to see how this can really work for the following reason: everyone is incentivised to pay the lowest fee possible for their required confirmation time (normally, one block). If you don’t really care about confirmation time the most cost-efficient place to be is right at the bottom of the queue but not actually falling off. This is a very tricky place to be. If you make a payment with your smartphone and then navigate away from your wallet app, it doesn’t know if someone came along 30 seconds later and kicked you out of the queue.

Even if it did wake up every so often to check, the Bitcoin protocol doesn’t let you know where you are in the queue ….. all you can see is whether your transaction appeared in the block chain or not. If you got completely evicted from the mempool, well, you have no way to know that. You’d just have to wait a really long time and eventually try again with a higher fee. Which, you know, the Bitcoin protocol also doesn’t currently support.

So setting up this kind of fluid fee market not only requires massive and complicated changes to Bitcoin Core, but also all the wallets, and the protocol too. That’s why it’s not easily deployed. Worse, if wallets pick some default fee level (very likely) then everyone just fights over that point in the queue and the outcome is essentially random — almost the same as what we’re shipping in 0.11B.

And anyway, the underlying logic is flawed. As the desktop Linux guys know, you can give something away for free and still not get popular. Bitcoin transactions have all kinds of costs other than miner fees, like the cost of exchanging them, the cost of learning to use the software, the cost of keeping them secure and the cost of the actual purchase itself. The only people who might generate infinite demand given zero costs are DoS attackers, and we have better ways to deal with them.

Q: Does this hurt future optimisations like IBLT?

No. Any future block propagation protocol has to handle the case where the mempools don’t match exactly anyway, as it already happens today. In a network where everyone’s mempool was entirely different, the ideal future protocol would degrade to just being the same as full block propagation like today.

But as the XT mempool limiting only kicks in during DoS attacks, during normal operation it should have no impact on optimised block relay protocols.