Uncovering a Game Of Stakes cartel

tl;dr: 53% of the voting power in game_of_stakes_3 is controlled by a single-entity cartel, and they’re going to be forked out. Fun times ahead!

Remove the cartel by hendrikhofstadt · Pull Request #263 · cosmos/game-of-stakes

This represents the first real-world test case of a 1/3+ cartel endangering a byzantine fault tolerant PoS network.

What happened? Read on.

Analyzing the Game Of Stakes submissions

Back on a rainy December day, during Game of Stakes signup, we forgot that the GoS was supposed to be an adverserial testnet, and built a GitHub bot which poked everyone who submitted invalid genesis transactions:

Now, having already downloaded and parsed all the submissions, the obvious next step was to do data analysis on them!

There were 180 valid submissions, give or take (2–3 PRs were structured such that our ingestion script failed to parse them, and were disregarded).

Here’s a graph of the number of GoS submissions over time:

Unless you manually edit the transaction before you sign it, the genesis transactions include the main IP address of the machine which generated it.

Here’s all IP addresses grouped by their autonomous system:

Cloud providers like GCE, AWS and Azure and Docker use internal IP ranges, which are classified as “Reserved AS, ZZ”.

Two things are immediately obvious —there’s a huge and unnatural spike of registrations, and a lot of genesis transactions originating from DigitalOcean.

We took a look at the raw data for the time range of the spike, and indeed, the vast majority of them were from DigitalOcean.

Oh boy! A cartel in the making?

GitHub helpfully provides the date of registration of any given user. We selected the subset of genesis submitters whose account is younger than seven days, and plotted their source ASN:

Gotcha! Apparently, the mysterious DigitalOcean submissions were also made using brand new GitHub accounts. This allows us to narrow down the set! It also suggests that there might be another group using a different playbook.

Filtering out the “Reserved AS” entries leaves us with 73 submissions from DigitalOcean whose GitHub account has been created in the past week. It’s reasonable to conclude that these represent one cohesive group, representing 40% of the total submissions. They would be able to execute a censorship attack right from the beginning of the GoS, without accumulating any stake (33+% is required for a chain halt/censorship attack).

Here’s the raw data of all DigitalOcean submissions that have a GitHub account age that were younger than 7 days on December 9, 2018:

We ask everyone to validate our findings and cross-check our analysis. If you look at the raw gist, you’ll find a detailed explanation for all columns.

Some notable findings:

• Most of the signups happened within a very short window, with regular spacing, indicating a highly coordinated effort, likely to be run by a single entity. It would be almost impossible to coordinate a group of 70 independently ran validators to submit their genesis transactions in such a coordinated fashion.
• The moniker names are mostly nonsensical and somewhat low-effort.
• The PR titles follow common patterns like very important upload or gentx upload.
• None of the GitHub accounts have a profile.
• All of the submissions were made using the GitHub web UI.
• All submissions were made from DigitalOcean VMs. None of them respond to ICMP pings (firewalled off or deleted).
• One moniker is bflabs, who are no strangers to collusion — having launched a fancy cartel collusion signup page — and have the necessary funds to pull off such a stunt. Of course, this could be a false flag — interpretation is left to the reader :-)

ಠ_ಠ

In Game of Stakes, there’s no cost to registering and each participant gets an equal stake. Therefore, the most obvious attack is a sybil attack on the registration process, which is why sybil registrations are against the rules:

Tendermint will disqualify and, if necessary, fork out players that undermine the goals of Game of Stakes to test interesting strategies for manipulating the incentive system. The most obvious reason for disqualification is trying to “win” through a sybil attack on the registration process.

It’s clear that someone successfully executed a sybil attack on the registration process. Registration required KYC, but it’s no secret that this can easily be sidestepped, like by simply paying real people to submit on your behalf.

After talking to other validators, it was agreed that this represents a major threat to the Game of Stakes network — the cartel could use their 1/3+ majority to execute a successful censorship attack against everyone else, winning GoS as the winning criteria is uptime (which censorship ruins), and that a swift community response and a fork would be necessary.

The intended defense against a 1/3+ cartel is a fork, coordinated off-chain, by the community. On-chain governance cannot be used for this, as any 1/3+ cartel can successfully veto any community proposal, and coordinating and pulling off such a fork is an important exercise for mainnet.

Ultimately, Game of Stakes is a scientific experiment in PoS research, and we all agreed that this would be an amazing thing to happen! For this reason, we decided to keep our discovery a secret and not ring any alarm bells….yet. The plan was to wait for them to make their move, then immediately announce a community fork once they started acting byzantine.

We didn’t want the cartel to know we knew, so they wouldn’t be discouraged from carrying out their attack, which would be boring.

We talked to Tendermint and they agreed with our plan, and didn’t remove their genesis transactions despite the rules violation so we could see a real censorship attack on a real network!

Waiting patiently

GoS launched in January, faster than anyone expected — turns out, launching a decentralized chain is a lot easier when 44% of the voting power comes online at once, which is exactly what happened! Seeing them come online all at once neatly confirmed our findings, and removed any remaining doubts.

We spent the next weeks collecting more evidence, checking and re-checking what we already had, building sophisticated censorship attack detection, sharpening our Layer 7 attack tooling and building tooling to filter accounts and validators out of a state dump. We went as far as building a fully automated robocall system that was going to wake up people once the censorship attacks started.

Zaki did his best to get them to attack:

A not-so-subtle hint

However, no such attacks materialized, not even when Aurel of Dokia Capital outright asked them to attack, already, please! :-)

What did materialize were transaction spam attacks which took down all 71 of the cartel’s (identical, apparently underpowered) DigitalOcean nodes, halting the chain, once again confirming our findings.

Whoever is running those validators is good at not getting jailed, which can’t be said about many others — by now, more than 60 validators got jailed, and correspondingly, the cartel’s voting power has increased from 40% to 53%, getting uncomfortably close to 66+%, while occupying most of the top spots for both stake and uptime.

Bonded stake on Game of Stakes 3. Cartel members marked in red.

Accumulated block misses in GoS 1, 2 and 3. Cartel marked red. Logarithmic scale.

With the Red Wedding hard fork coming up, it was increasingly unlikely that the cartel was going to do a cartel-based censorship attack, so after a discussion with fellow validators, it was decided to propose a community fork once the genesis file for the upgrade was announced.

We’re going to submit a pull request with a new genesis file for the chain IDgame_of_stakes_5, based on the official game_of_stakes_4. The genesis file will not contain the cartel validators, and their funds will be transferred to the community pool. All tooling used for filtering the state as well as evidence will be published for the community to examine.

What evidence is there?

• Genesis transaction analysis, as detailed above. The analysis can be replicated using the data in the game-of-stakes repo.
• Prevote timing during each chain start, as seen in the logs.
• Lack of prevotes during the spam transaction chain halt. The set of missing validators during the halt perfectly matched with our previous findings.
• The cartel’s fee patterns changed multiple times during Game of Stakes.
• The cartel’s autobonding pattern changed during Game of Stakes, switching from separate Withdraw/Delegate transactions to combined transactions, as we had been doing for a long time. This suggest that all of their nodes run identical software.

Voting pattern findings can be replicated by anyone who savedconsensus:info-level logs. Autobonding and fee patterns are persisted on the game_of_stakes_3 chain (don’t delete your data directories!).

Why fork them out?

• Concentrating 53% of the voting power in a single, densely connected network running with a single provider results in highly unrealistic network behavior. GoS will be more valuable as a real-world testnet without such an unnatural voting power distribution.
• This was evidenced by the spam transaction-induced network halt, which was caused by them crashing all at once.
• It also sped up the decentralized start — the network launched when they brought their nodes online.
• As quoted above, doing a sybil attack against the registration — while really cool from a game theory point of view — is not the intended way to win Game of Stakes. Not forking them out would damage the credibility of the Cosmos project, which isn’t in anyone’s interest.