A Guide to Effectively Cheating NFT Launches (and detecting cheaters)

Honest Protocol
11 min readOct 25, 2021

--

Authors: Maximino DiGiacomo-Castillo, Kun Peng, Ricardo Rosales, Nicolas A. Bax

Contents

  1. Introduction
  2. How to use our tools to cheat at NFT launches
  3. Detecting “Anomalously Lucky” NFT buyers
  4. More shenanigans (other irregularities we observed)
  5. How can we level the playing field?
  6. Join us

We’re NFT fanatics. We believe that NFTs can revolutionize the art and collectibles spaces.

As described in Paradigm’s Guide to Designing Effective NFT Launches, NFT launches are often unfair to the average user. Hasu and Anish Agnihotri describe several pitfalls which can give sophisticated users an edge and allow such users to extract value from both creators and other collectors.

One common design error is what Hasu and Anish Agnihotri term “exploitable fairness”. Part of the fun for buyers of NFTs is the “gacha game” aspect: buyers may get a common NFT, or if they’re lucky, they may get an NFT with rare traits, which can sell for several fold more than their common (“floor”) counterparts. To be fair, NFTs should be minted randomly. However, sophisticated users are constantly on the lookout for ways to increase their odds of minting or buying a rare NFT.

We have developed and deployed tools for the purpose of gaining an advantage when buying NFTs and we aren’t the only ones; we’ve observed others using such tools for months and recently numerous paid services such as traitsniper.com have appeared. Today, we’re open sourcing our tools, describing how we use them, and how we can detect when someone else has used similar methods to gain an advantage in an NFT launch.

Our cheating tools work by finding metadata before it is available to the average user, and then using it to acquire more rare NFTs during the minting process, or by buying unrevealed NFTs before the metadata is uploaded on OpenSea. Similar cheating can also be carried out by project insiders.

We detected a lot of cheating using methods similar to ours and some instances are described here. Our hope is that spreading awareness about such tactics will prompt the NFT community to demand provably fair NFT drops.

How to use our tools to cheat at NFT launches

Step 1. Acquire metadata

First, you’ll need to find the token metadata for the collection you’re targeting. Check the token’s URI to see if it is pointing to an IPFS page or website with metadata. If it is, we’ve provided a script called pulling.py that takes the contract address as an input and then rapidly scrapes all of the metadata for an NFT collection.

Our pulling script takes the ERC-721 contract address as an input and then rapidly scrapes the metadata.

If you can find the metadata before the initial sale is complete, then you’re golden — just proceed to step 2 to identify which NFTs are worth minting, and then try to mint those.

We’ve observed many different mechanisms that projects use to hide their metadata. The most common is to hide the metadata until after an NFT launch is complete and then update the metadata on the project’s website. In other cases, the metadata is stored on IPFS and updated by changing the base token URI. We’ve had some success finding leaked metadata by querying OpenSea’s API or by scouring testnets and Arweave, where devs occasionally leak the IPFS hash for their metadata. Even if this method doesn’t yield metadata, it can provide other valuable insights.

A sophisticated user describes the process they used to analyze an NFT project prior to its launch. (No metadata was leaked in this example.)

Sometimes we were unable to find the full metadata but were able to find leaking images. In this scenario, we just use a “for” loop to download all of the images, and then look through them manually for traits that we know are rare.

A simple for loop can be used to scrape leaking NFT images before minting and/or reveal.

We have also observed instances where we were unable to find leaking metadata, but someone else was still able to gain an edge in minting. There are numerous possible explanations for this observation. One is that we were unable to find publicly leaked metadata whereas someone else was. An alternative explanation is that an insider possessed information — insiders may be directly related to a project, or they could be a 3rd party that is hosting the project’s metadata.

Step 2. Calculate Rarity

There is a strong correlation between NFT price and perceived “rarity.” The most widely used website for NFT rarity is rarity.tools, and they use the same algorithm to generate rarity ranks for almost every collection.

Sale price as a function of rarity rank for the last 200 MekaVerse NFTs sold. (As of 10/26/2021)

Our rarity.py tool uses the metadata from step 1 as an input and then outputs the rarity score and rank for each NFT in a collection.

One thing to look out for is that the rarity score methodology does differ slightly; for some collections, rarity.tools uses “number of traits” as an attribute, but for others it doesn’t. Some collections also have custom rarity scores which will need to be scraped separately. Now that you have the rarity for every NFT in a collection, go on to step 3.

Note: we dislike rarity.tools’ ranking methodology but that’s a subject for a different article.

We noticed that early on during NFT summer, a lot of NFT minters/buyers that we think were anomalously lucky weren’t using calculated rarity scores. Instead, we believe that they were simply scraping the metadata and looking for NFTs with a single very rare trait, as opposed to combinations of somewhat rare traits which together made an NFT rare. As time progressed, we noticed that some of the buyers began calculating rarity scores as described here.

Step 3. Minting or buying

Now that you have rarity data for the entire collection, it’s time to buy. If the collection is still in the minting phase, then you can try to mint the NFTs that you know are rare. To do this, check Etherscan manually, or better yet, write a bot that gets the tokenID of the last NFT that was minted. When the next NFT to be minted is the one you want, just send your minting transaction. Depending on the rate at which minting is occurring, this can be done manually or you may need to use a bot to time your transaction just right.

Because most launches allow batch minting, you may need to mint multiple common NFTs in order to snag a rare one. In some cases, you may also find multiple rare NFTs that you can mint in the same transaction, particularly when the NFTs aren’t randomly distributed.

This anomalously lucky user minted 5 BlazedCats. The 5th was rare and could be sold for a price that easily offset the cost of the other 4.

Since you know you’ll profit off the rare NFT, you can sell the other ones at or even below what you paid to mint them.

There is also one well-described example where someone used a smart contract, along with flashbots, to guarantee that they only minted rare NFTs.

Step 4. Pre-reveal Sniping

Even if the mint phase of an NFT launch is concluded, or you weren’t able to obtain leaked metadata, you can still gain an edge in the next phase of the NFT launch. Many projects reveal their metadata after minting has concluded. However, there is a delay between when metadata becomes available and when OpenSea has finished updating. During this period, you can rapidly run the scripts we provided to obtain metadata and rarity scores and buy any rare NFTs that people listed for sale prior to reveal.

Detecting “Anomalously Lucky” NFT Buyers

Our tools can also be used to detect buyers that we call “anomalously lucky.” To do this, you’ll need rarity data for an NFT collection, as well as a list of NFTs minted by each buyer.

We’ve provided rarity data for some projects on github or you can use our tools to get rarity data for other projects.

Once you have rarity data, the next step is to collect minting data for a collection as shown in find_minting_data.ipynb. This will give you a database of every NFT minted by each account, and its rarity rank.

Every collection is different and you may need to tweak which statistical tests you’re using but we’ve provided some basic tools and data to help you get started.

A good first start is to run a Kolmogorov–Smirnov (K-S) test on the minting database — we have an example of this in ks_test.ipynb.

The “null hypothesis” that our K-S test is attempting to reject is that a user minted NFTs randomly. The K-S test checks whether the rarity rank of all NFTs minted by a user may arise from a uniform distribution. In certain cases, such as when NFTs weren’t randomly distributed or part of the collection was gifted instead of minted, the probabilities get more complicated and you may want to generate synthetic datasets which consist of the NFTs an account should have had a chance to buy.

A histogram showing the rarity ranks of DogeX NFTs minted by an anomalously-lucky user (red), and an average user (blue).

Our K-S test code will run through every account that minted multiple NFTs from a collection and outputs some data about the lucky accounts. Our code can also be modified to identify accounts which bought minted NFTs before the metadata reveal.

Example output for the two luckiest accounts that minted DogeX NFTs.
Blue dots represent each NFT in the DogeX collection. Red dots represent DogeX NFTs minted by an anomalously lucky account.

Interpreting the output of these statistical tests is nuanced. A p-value of 1E-5 does not guarantee that an account was grifting — if looking across hundreds of thousands of accounts, we would expect to occasionally find people who are that lucky. (A p-value of 1E-22 on the other hand…)

We also took a close look at several instances where members of the NFT community alleged that an account had grifted, and found that the accounts being accused were just slightly lucky (1% or 0.1% chance of occurring, absent additional non-statistical or off-chain evidence).

The K-S test may also give low p-values if the NFT collection itself wasn’t truly random. Conversely, just because an account doesn’t have an anomalously low probability also doesn’t mean that an account wasn’t grifting. In fact, we are confident that most of the accounts we used would not be identifiable by any statistical test, and we definitely aren’t the only ones clever enough to remain undetected.

More Shenanigans

We used our scripts to take a close look at many NFT drops. Most showed no signs of anomalies and probably were executed in a fair manner (although not provably so!) However, during the course of our research we observed several other types of anomalies.

Non-random distribution of traits

As alluded to by Hasu and Anish Agnihotri, we also observed many NFT collections that did not have a random distribution of rarities. In certain cases, this means that sophisticated buyers can increase their odds of minting a rare NFT, even if they can’t guarantee the exact number they will receive because minting is occurring quickly.

Great Ape Society Bananas showed an unusual distribution of rarity ranks.
Quaks had non-random rarity distribution. As a result, most users who minted towards the end of the launch would appear to be anomalously lucky.

Non-Standard Minting Processes

We observed numerous NFT launches where the minting process for rare NFTs was unusual and randomness was impossible to verify. For example, one project made unverifiable claims that users were randomly selected off-chain to receive the rarest NFTs in their collection. These NFTs had a significantly higher resale value than the remainder of the collection, which was minted “first come, first serve”.

Some projects have a special minting process for rare NFTs.
A Discord message from an NFT project leader which describes how they randomly selected accounts to airdrop the most valuable NFTs to, along with instructions for verifying that the airdrop was fair.

We use a metric that we call the “first flip” metric to gauge how profitable minting and reselling is for a given collection. In this particular collection, about 50% of the total profit came from the NFTs that were distributed using the random selection process described in the Discord screenshot above. Note: This metric can’t rule out wash trading.

Screenshot showing the profit from reselling 8 Bit Universe NFTs after mint by token ID. About half of the first flip profits came from token IDs 1–100, which were given out after minting concluded.

Unverifiable Commits

We observed several projects that claimed to be using a provably random commitment scheme to select NFT metadata where we were unable to find the commitment prior to token minting, nor were we able to verify that the metadata was generated after minting had concluded.

Whenever a launch does not use a commitment-reveal scheme, it is impossible to rule out that project insiders are not changing metadata of NFTs they minted or bought on the post-secondary market.

How can we level the playing field?

We want the NFT space to succeed but we think that the shenanigans we have observed are extracting value from most users which is detrimental to the long-term health of the ecosystem.

  1. Provable fairness

Cryptographic schemes for provably fair NFT drops already exist and we have already seen some projects utilize them. We hope that by spreading awareness about information asymmetries in NFT metadata, consumers will demand provable fairness from projects they patronize.

We would be happy to audit NFT launches with regards to randomness but ultimately we want consumers to learn about the magic of cryptography.

2. Fair metadata reveals

Rather than relying on third parties such as OpenSea and rarity.tools to provide users with NFT metadata, projects should make NFT metadata available themselves immediately after it is revealed, thus negating part of the edge from scripts such as our pulling.py.

3. Enough with the “first come, first serve” NFT launches

As described in the Paradigm article, “first come, first serve” launches disadvantage the average user and revenue that could be going to artists and projects is instead going to miners. We didn’t go into it much in this article but we agree with Paradigm —please stop doing your launches that way!

Join Us

What we’ve shown here is just the tip of the iceberg. We have plenty of ideas for other methods to identify cheaters and are eager to hear ideas from the community. If you’re interested in joining our mission to research suspicious activity, drop by our Discord channel. To incentivize contributors, we’re setting up a bounty fund which we’ve seeded with our profits from sniping NFTs with leaked metadata. If you just want to follow along, follow HonestNFT on Twitter and check out our new project’s website.

edited on 10/27/21 to add labels to axes on two graphs.

edited on 12/5/21 to make graphs higher quality.

edited on 12/15/21 to utilize relative frequency for the DogeX plot

We cross posted this article on Mirror here.

--

--