To Catch a Thief

Hunting for an attacker leads to a Grin bug

At the end of April, something was wrong at Grinmint. Our pool luck was terrible and our calculations for profitability weren’t adding up. We estimated that around 10% of our pool hashrate was off. Intuitively, we knew we either had a share accounting bug…or there was a thief in the midst of our pool.

Finding an exploit in a mining pool — whether it’s hunting for an elusive bug or a crafty cheater — is an extremely difficult process. You have to monitor several indicators and cross-correlate them in order to identify that something is going on. Even then, it can be hard to determine whether it’s a string of bad luck or how the attacker is managing to pull it off.

During the last week of April, after an extensive investigation, we were still unable to find either the bug or the attacker, but we were becoming more and more suspicious that it was an attack that was faking hashrate.

The Investigation

There are several ways to fake hashrate. The most common one is to submit duplicate shares by sending a share multiple times. This is a classic attack and easy to countermeasure.

The second type of attack is called block withholding. Withholding blocks is basically a strategy to cause losses on a mining pool: you mine normally and submit shares, but every time you find a share that is a block you do not send it to the pool. This strategy is detrimental for the miner who loses some profits, but the biggest loser is the mining pool and all its miners as they will get less than expected. A way to monitor for a withholding attack is to mark the users that found a block. And so we did.

On May 6, after implementing this user marking verification, I took a look at the user/hashrate/block found table. To my surprise, one of the miners on Grinmint never found a block for a period of approximately two days. While possible for a very short period of time, it’s highly unlikely for that long of a period.

The attacker was identified.

The Exploit

Having found the attacker, it was still too early to determine whether the attacker was withholding blocks or had found an exploit.

We had to catch him red-handed in order to understand his exploit and so we began to log every share this user was sending. After a few minutes we got the following:

WARN[2019–05–03T19:34:42Z] WORKER: Received hacker share” addr=”X.X.X.X:XXXX” height=155179 jobID=0 nonce=16620723100425694634 pow=”[81466247 136239418 169906556 174870950 305298631 316454817 371131270 377206933 385175416 523906182]” userID=16024

The log shows that the attacker’s shares had a cuckoo cycle length of 10, instead of the required 42 (you can see this yourself by counting the number groups in the pow="..." array)

Invalid Shares

The rest of this post is a bit technical and so I highly recommend you read An Introduction to Grin Proof-of-Work to understand what follows.

The above log shows that the pow “..” parameter array has a length of 10. Meaning that the miner was sending a valid share, but much less than the required 42 cycle share. How was this possible? At Grinmint, we use our own open-source library, Libgrin, for share verification and validation. My first thought was that I made a mistake in the share verification, but I wanted to double check if Grin also accepted it. I looked at our Grin stratum logs:

0190503 21:04:50.800 INFO grin_servers::mining::stratumserver — (Server ID: 0) Got share at height 155179, hash 00415e6ba476, edge_bits 29, nonce 16620723100425694634, job_id 0, difficulty 1257206/2914935738, submitted by white512

A Grin Bug

What I had thought was a bug in our share validation library, Libgrin was actually a bug in Grin. I tested this by writing a proof-of-concept on both Libgrin and Grin and verified that this kind of invalid share was getting accepted.

At this point, the problem was no longer an attack on Grinmint. My biggest fear was a consensus-breaking bug in the Grin core protocol. That someone could mine and propagate a block with a shorter cycle length would represent a major flaw in Grin’s proof-of-work: you would basically be able to trick the system by spending less than the required work to find a block.

I had to confirm this with my fellow Grin developers. After several verifications we found the issue in the Grin code (which Grinmint’s Libgrin had copied). The good news was any block mined with a lower cycle would be rejected by the network. The bad news was all mining pools were vulnerable to the attack. We needed to make sure all affected parties had a chance to close the attack vector before another person could exploit it.

Once we were sure that this vulnerability did not allow anyone to mine a block, we communicated with the other mining pools and told them how to fix it. A fix was subsequently released on Grin master branch and in Libgrin.

Immediately after we patched Grinmint to refuse any share with a shorter cycle length, our profitability returned to the expected range. Not only had we found a Grin vulnerability, fixed it, but we did the right thing by reporting it.

We at BlockCypher are committed to providing more transparency in the mining pool ecosystem.

For questions and to be notified on our latest updates, follow Grinmint on Twitter @Grinmint1 and join us on Telegram https://t.me/Grinmint