Detecting NFT Hacks With No Machine Learning
TL;DR: In this article, I share the latest findings of the ZenGo wallet research team that just uncovered more than $24M worth of NFTs stolen in offline signature attacks with our NFT Hack Detectooor 💥.
Using these findings, we were able to alert in advance on the attackers’ account that stole millions of USD worth of NFTs from an industry-leading NFT pioneer.
In this blog post, we’ll share the details of our abstract scam detection logic as well as a few insights we’ve found along the way involving North Korean attackers.
Offline Signatures: A Quick Recap
When interacting with smart contracts you might be required to sign messages. This allows your wallet to authenticate you vis-a-vis the dApp with which you’re interacting.
Let’s imagine you own an NFT and you want to sell it. Typically, you would sell it through the OpenSea Seaport contract, requiring you to sign an offline message with a parameter indicating the amount you’re willing to sell your NFT for. Now, let’s say a scammer makes you sign a fraudulent listing where this parameter is empty — not hard given how cumbersome it is to understand the contents of the signature (see image above). Once the scammer has your signature they can initiate a manipulated transaction with your signature (using, for example, the ethers.js library) allowing them to take your NFT for free.
Uncovering Scams: Our Methodology
The Tool: Dune Analytics
Dune Analytics provides a powerful query engine as well as very cool abstractions allowing you to write powerful queries on the blockchain data.
Luckily for us, Dune community members have already created a table to monitor all OpenSea marketplace trades, and all we had to do was to express our abstract detection rule idea in a SQL query on this table.
SELECT * FROM nft.tradesThe Idea: Creating the Trade Ratio
No rocket science here, the idea behind our abstract detection rule is very simple.
If a valuable NFT was traded through OpenSea for a suspiciously low amount, then it’s likely to be a scam.
In order to measure how far an NFT was traded from its estimated floor price, we created a metric called the floor ratio.
In the distribution below we observe that the Trade Ratio are distributed around 1. This is expected because most of the NFTs are traded around their floor price. However, see the bump close to 0? Those are NFTs traded for close to 0% of the floor prices and it looks like they are likely to be either hacks or errors.
The Floor Price: Measurement
Opensea, the largest NFT marketplace, defined a standard for measuring floor prices: Take the lowest of the latest trade price for a given collection.
When querying large amounts of historical data — where reproducing historical behaviour is computationally expensive — we define the floor price differently: We look at the distribution of all the trades of a given collection in a given week and take the 10th percentile.
floor_prices AS (
-- CTE to get the weekly floor price of nft collections.
select
date_trunc('week', block_time) AS date,
nft_contract_address,
count(*) as n_trades,
percentile_cont(.1) within group (order by amount_original) as floor_price_original
from nft.trades
where true
and currency_symbol IN ('WETH', 'ETH') -- todo: support all changes
group by 1, 2
)The Popular NFT Projects
In order to be confident in our floor price measurement, and thus in our ability to flag scams, we only included the top 1000 NFT collections in terms of historical trading volume. Let’s see why.
Take a look at two very different NFT projects, Bored Ape Yacht Club and Keepers of the Inn. While both are legit collections with interested community members, the latter has very little weekly activity (see chart below).
This small sample size (statistically speaking) results in noisy floor prices.
The Accidental NFT Sales
When the transaction is sent from the seller, it could be an ill-treat (for example sending a ridiculously low bid to NFT holders hoping they’d inadvertently accept them) but not necessarily of offline signatures. Therefore we excluded them from our stats, although they may represent an attack (for example, as described by this hypothesis of users clicking by mistake).
The Final Logic: Balancing Precision and Recall
As described above, the main idea is to include trades for which the trade ratio is below 0.01. Then we add multiple conditions to eliminate legit NFT trades (this is part of our effort of trading off some recall in favour of precision).
select *
from trades
where true
-- scam filtering heuristic
and floor_price_crypto > 0.1 -- high floor value
and trade_price_crypto <= 0.01 -- low purchase amounts (higher likelihood of being a scam)
and trade_ratio <= 0.01 -- low purchase to floor value ratio
and get_ens(buyer_original) is null -- ens buyers are less likely to be fraudsters
and not is_seller_originating_tx -- those are likely to be offers accepted by the seller, thus not scams but potential mistakes
and n_past_transfers = 0 -- remove trades where seller transfered nft(s) to the buyer in the pastThe Performance: Evaluation
Given the freshness of the offline signature scheme, no available dataset of scam transactions could be relied on to perform this analysis. One of the ways we validated our precision was by manually checking each of our 25 top attacking addresses on OpenSea.
At the time of writing, 19 out of 25 addresses were indeed suspended on OpenSea’s website giving a proxy of precision close to 76%.
Some Insights 🤯
Insight 1: $24M worth of NFTs stolen since January 2021
As we were very conservative in our estimations (e.g., taking into account only high-priced NFTs, using floor price as an estimation for NFT value, preferring to be over-restrictive than over-permissive when including a transaction in our dataset), we believe that our estimation of NFT theft can be considered a conservative estimate of the true volume of this phenomena.
Insight 2: See the spike in February 2022?
This is a phishing attack that took place on OpenSea and that was later reported by The Verge.
Insight 3: The North Korean Connection
Since August 2022, we notice an increase in offline signatures attacks.
Compared to the preceding period, this attack shows a much higher volume of NFTs stolen although the average value of each NFT has significantly declined.
To use the very appropriate “fishing” methodology: While previous attackers were spear-phishing by selectively targeting a few high net-worth victims and attacking them, this new campaign appears as an industrialized mass dragnet phishing: Targeting a much wider audience of victims and non-selectively catching small and big fish.
The breakthrough came when we were able to correlate one of the leading attackers to another threat intelligence report by SlowMist. Last December, SlowMist released a report claiming North Korea was behind many NFT phishing attacks. They were able to get to such attribution by following the infrastructure of the phishing websites and connecting them to other known North Korean resources.
Insight 4: Moonbird creator Kevin Rose hacked.
He is the #2 all-time victim of this scheme since January 2021.
Conclusion
We hope that the NFT Hack Detectooor will help the community better understand and quantify the issue of scam offline signatures. We’re committed to providing users with accurate and up-to-date pricing information, and we look forward to continuing to improve the precision of our tool while expanding our coverage.
👀 In the meantime, go see if you’ve ever been hacked using our dashboard: https://dune.com/zengowallet/zengo-nft-hack-detectooor
📱 And also, be safe and store your crypto and NFTs on ZenGo.
