Proof-of-Stake Validator Sniping Research

The Merge is fast approaching. In a few months, the Ethereum chain will switch from proof of work (PoW) to proof of stake (PoS). This is an exciting development, but we need to highlight an important privacy issue with the post-Merge chain. The HOPR team has been leading research into the functionally identical Gnosis Beacon Chain (GBC) in collaboration with the Decentralized Systems and Security Group at Imperial College. We’ve identified large perverse incentives for validators to conduct sniping attacks on each other in an attempt to poach their block rewards.

This is possible because the PoS validation scheme requires validator public keys on the beacon chain to be known to everyone ahead of time, so everyone knows the validator ordering for slots within an epoch. This foreknowledge, combined with information about a validator’s IP address, enables a particularly nasty attack whereby someone can deliberately take down a validator’s node — e.g., via a denial of service (DoS) attack — and the next validator can poach their rewards.

This possibility has been known for a while — in fact it’s mentioned in several Ethereum security audits from 2020 (here’s one from Least Authority and another from Quantstamp). But it has generally been dismissed as low severity.

However, the research conducted at HOPR over the past few months, as well as parallel research from our collaborators in the Decentralized Systems and Security Group at Imperial College London, lead us to conclude that the risk is much higher, for three reasons:

1. The incentives to conduct this attack are much higher than previously thought
2. It’s easier to identify validator IPs than previously thought
3. The attack itself is easier to conduct and harder to mitigate than previously thought

Let’s outline the steps of the attack and then look at these three points in more detail.

Validator Sniping — the Basics

The basic outline of the attack is simple: watch the beacon chain network for long enough to form a link between a validator’s public key and their IP address, then when that validator becomes block proposer, take their machine offline with a DoS attack. They will be unable to complete their block proposer duties and their slot will expire. The role will pass to the next validator, who will be able to take all the MEV opportunities and transaction fees the sniped validator had access to.

But how would you get the public key and IP address information needed to conduct the attack? To understand, we need to look deeper into how proof of stake will work in ETH2 and equivalent setups.

Every epoch, a set of validators is randomly selected to propose blocks, and every validator gets to validate one block within the epoch. This validation process is conducted within these small groups, but the confirmations of each step — known as attestations — are broadcast throughout the network. This all happens publicly, and the only requirement for running a validator node is that you put up the requisite stake. Therefore, anyone can pay to have access to all this data.

As you run your validator node, you’ll gain information about other node runners’ IP addresses and public keys. This link isn’t straightforward — attestations are often bundled for efficiency, and nodes won’t be connected to all other peers in the network, making for incomplete data and a lot of noise. But we’ll show below how it’s possible to cut through this and link IP addresses to public keys with a very high degree of certainty.

Once you’ve established this link, all that remains is to wait for a suitable attack opportunity. You can then launch an extremely targeted denial of service attack (DoS) on the target validator. This isn’t like the distributed denial of service (DDoS) attacks which we’re more familiar with that take down major services for hours. It wouldn’t need to be that intense or long-lived: it just needs to knock the target machine out of action for long enough that they can’t fulfill their slot duties. But what can be gained here, and how often do these attack opportunities occur?

It’s here that previous research seems to have misunderstood the stakes.

Mismatched Incentives

Our assessment is that when previous researchers and auditors considered this attack, they were mostly thinking about the incentive to cause chaos: to indiscriminately disrupt the chain, or to cause validators to be slashed by blocking them from fulfilling their validation duties.

What’s been missed is that the colossal rise in MEV creates direct incentives for validators to attack each other, either working alone or in collaboration with a dedicated validator sniper who would look for opportunities to take down particular machines, much like how MEV searchers operate in tandem with miners / validators.

For an attack to be viable, three things must happen simultaneously:

• Knowing the IP address of an upcoming block proposer
• Having control of the next slot (either because you’re the validator in that slot, or you’re able to collaborate with them)
• Sufficient MEV opportunities in the mempool to be worth poaching

Our assessment is that this isn’t some once in a blue moon set of circumstances. In fact, we’d expect to see this happen very frequently. This is because:

• Finding validator IP addresses with high confidence turns out to be fairly trivial
• The frequency of collaborations between MEV searchers and miners shows that opportunities are vast and these interactions emerge with little friction in the DeFi space. Although DoS-facilitated MEV attacks would be more widely frowned upon than vanilla MEV, which exists in an ethical grey zone, we’d still expect similar collaborations and services to arise quickly.
• MEV opportunities occur very frequently in Ethereum at present, and there’s no reason to think the switch to ETH2 will change this significantly

Linking IP addresses to Public Keys

It has long been known that it could be possible to watch the network and form links between validators’ public keys and IP addresses. But as far as we know, HOPR is the first project to actually try it. The next sections explain in detail what we did and how easy it was.

Our Setup

Since April 21st 2022, 11 HOPR team members around the world have been running validator nodes on Gnosis Beacon Chain, using hardware nodes from our great partners at AVADO. They voluntarily shared their IP addresses and public keys with our analysis team so that the results could be confirmed. Note that the IP addresses were not used in the analysis process except for verifying the results.

Separately, we ran another node which was lightly modified to harvest the IP addresses of all connected peers when they broadcast attestations and dump this to a database of IPs and public keys using Elastic Stack. The node modification was fairly trivial, as the Lighthouse client facilitates validator monitoring as standard. (This feature will be important later too, when we discuss ways to improve this data harvesting.)

This generated a LOT of data. For example, between 1st June and 1st July we stored 2,285,214,572 IP / public key pairs, which could be associated with 1,253 unique IP addresses.

Linking Public Keys to IP Addresses

So how did we turn this huge amount of data into a confident link between IP addresses and public keys? Remember, just because our node received information about an attestation linked to a public key from a particular IP address, that does NOT mean that this is the IP address of the validator node with that public key. This is because some nodes aggregate attestations for faster dissemination throughout the network. In fact, our node didn’t even register that these were aggregated attestations, instead logging them in the same way as single attestations.

So the task is to wade through all this noise.

First, we pick some public keys to try and link. For this experiment we used the public keys which we already knew from our testers. It’s crucial to understand that this step is valid because at no point did we use our foreknowledge of the IP addresses to establish the link. The only reason to pick these public keys in advance was because we knew that we could validate these results. We could equally have just picked random public keys and worked through them one by one, eventually hitting the ones controlled by the HOPR team.

Even narrowing down to these 11 public keys, it’s still a lot of data. From 21st April to 1st July, our node gathered 431,639 pairs of IP addresses associated with the 11 HOPR public keys. Over this timeframe, 1613 unique IP addresses were associated with the 11 HOPR public keys. But which were the true ones?

The first thing to do is divide the data into discrete chunks of time. The choice of timeframe is arbitrary, but we chose a day because this seemed like it would give a manageable amount of data. For example, if you only looked at the past 15 minutes of attestations we might not be connected to a given HOPR node. Therefore, you would not have any information to match the public key with the correct IP address.

The figure below shows the attestation count by IP and pub key of HOPR Team GBC nodes throughout June, with attestations separated by day.

As we can see, our harvesting node did not connect equally to all HOPR nodes. In fact some days it connected to hardly any of them. This makes sense as a node cannot be connected to all 1600 nodes in the network at all times, and nodes are not always consistently up and reachable.

Still, this is enough to give a strong link. For a given day and public key, we simply looked at which IP address was most frequently associated with attestations from that public key. This is really the most basic approach possible, but even this we found produced the correct pairing in 62% of cases.

But we can easily do better. This method does nothing to account for the fact that there will be days when our harvesting node will not connect directly to the node we’re trying to identify. And put the other way, it doesn’t sufficiently use the fact that when a given HOPR pub key is associated with the correct HOPR IP on a given day, then the HOPR IP produces the highest attestation count among all other IP’s associated with that particular HOPR pub key in 99% of all cases.

To improve this, we used a reweighting method where we reweight the percentage share of a given IP from the first approach with the total number of associated attestations across all days.

In essence, the weighting downweights days in which a given HOPR node is not connected and shifts this weight to days where a given HOPR node is connected.

This approach increased the probability of identifying the correct IP address from 62% to 89%. It also reduced the set of possible IP addresses associated with each node to 1–3.

So establishing a reliable link from public key to IP address requires data to be gathered over quite a long period. But how long? The figure shows that we needed to collect around three weeks of raw attestations to link all 11 of our nodes. But this was because we already had a set of public keys we were interested in to prove our proof of concept. If we just wanted to find ANY public key to snipe, and we were prepared to accept a lower certainty threshold, then a lot less time might be needed.

There are various other techniques which could cut this time by a lot, such as:

• Running multiple validator nodes
• Cutting the connection to nodes whose links have already been established to focus on unlinked nodes

This last one brings us back to the Lighthouse modifications mentioned above. Instead of connecting indiscriminately to nodes, you can choose particular validators to connect to or ignore. This is usually used for diagnostic purposes, to test the health of nodes you control, but this can be subverted to facilitate the attack.

One limitation is that there’s a lot of redundancy in the dataset. Since we’re only interested in building the IP address database, not facilitating validation on the network, there’s no point in gathering data from nodes where we already know the IP / pubkey link. So if we set a confidence threshold beyond which we’re happy that the link has been made, we can deliberately drop connections to those validators to instead focus on currently unlinked nodes. In this way we can more efficiently build up a picture of the whole network, populating the IP address database much more rapidly.

Attack Feasibility

Of course once you have the IP / pubkey link, you still need to carry out the attack. How feasible is this? That’s what our research partners from Arthur Gervaise’s group at Imperial College London are currently trying to determine. We expect their research to be published in the next few months, but we can already make some assessments.

First, this doesn’t need to be a large-scale attack. The target machine just needs to be taken down for a minute or so, long enough for the slot to expire.

Second, client-level solutions don’t seem to be much help here. There were a few proposed solutions in the Teku audit from 2020, but they only work if the attacker is trying to overload the node, not the whole machine.

Third, we’ve seen much larger scale DoS attacks brought to bear on crypto in recent months, for potentially smaller rewards. It seems inconceivable that people wouldn’t take the opportunity to conduct a simpler, less detectable attack for higher returns.

Next Steps

So what can be done about it?

The most obvious short-term solution is to manually protect your IP address. Good VPN hygiene is important of course, but note that this likely isn’t enough to protect you. Quite apart from the fact that many VPN business models open you up to other privacy issues, VPNs simply don’t provide enough IP address protection. As a case in point, let’s return to the geolocation diagram from before:

We don’t want to doxx our own team members, but let’s just say that this map accurately locates the IP addresses of the nodes, but not their physical location…

So you need to cycle your IP address fairly rapidly. How rapidly? That depends on how fast the IP link can be established. We needed weeks with our naive approach, but we were just running one harvesting node with little optimization. Estimates from collaborators suggest that time can be reduced as low as 15 minutes fairly easily, especially if you’re happy to accept a modest hit to the confidence threshold.

Obviously as transport-level privacy project, we’re going to endorse a transport-level privacy solution. Attestations sent through a mixnet like HOPR’s would show a different IP address basically every time (the IP address of the penultimate hop in the relay chain). Attestations are also small enough that only one HOPR message would be required, so there shouldn’t be any appreciable latency or packet loss.

This solution would also keep validation as a viable option for home validators, which we feel is a vital part of the decentralized future. We’ve been delighted to see that our friends at Gnosis Chain feel the same, frequently rejecting complex or resource-intensive solutions which would add further barriers to becoming a validator.

But the reality of course is that a variety of solutions should be adopted, across every layer. The blockchain is the fundamental source of truth that underpins everything we build on top of it — no effort is too small to ensure its integrity and security.

Responsible Disclosure

Finally, a note on responsible disclosure. One of the biggest criticisms of our initial Twitter threads is that we should have released this information less publicly, through the private disclosure channels used for severe bugs and other threats.

While we agree the initial tone of our threads was suboptimal, we disagree that this is an issue for behind closed doors. This is something which has been known about for several years, with no sign that it’s being taken very seriously. No funds are currently at risk, so there’s no immediate danger of signal boosting this information.

This is an issue for everyone, but validators in particular, and we feel it’s vital that everyone be included in the conversation. We look forward to collaborating with as many people as possible on this issue in the coming months. If you’d like to reach out to us, please use one of the links below.

Sebastian Bürgel,
HOPR Founder

Website: https://www.hoprnet.org
Twitter: https://twitter.com/hoprnet
Telegram: https://t.me/hoprnet
Discord: https://discord.gg/dEAWC4G

Forum: https://forum.hoprnet.org
Staking: https://stake.hoprnet.org
Bounties: https://bounties.hoprnet.org