A minute later, Alex was there to the punch.
“Don’t scare me like that.”
“This is not a drill.”
There are some things you don’t want to hear. “U up?” is one of them. It either means your friend is messing with you, or your smart contract is live on mainnet with a gaping hole in it, ready for the TVL to be drained at any moment by some blackhat with nothing better to do. Tough luck. DeFi is an unforgiving jungle.
This was the latter, and it kicked off a fast-paced, nerve-wracking war room that would last for the next 48 hours and bring members of the DeFi security community together to save Primitive Finance.
The reality was that Primitive Finance did have a gaping hole, a critical bug in its smart contract. Earlier on Friday, the Dedaub team, led by Yannis Smaragdakis and Neville Grech, had been poring through the code after their automated scanner flagged some lines.
Yannis and Neville, both academic computer scientists by training, started investigating the warnings. By Saturday, it became clear that they had something real — a vulnerability that effectively allowed a malicious user to create a fake token and swap that token for a user’s real tokens, as detailed more extensively in the Primitive Finance technical postmortem.
Everything was at risk.
In technical terms, the attack vector is called a confused deputy attack. You have a deputy, an intermediary, that is a trusted entity as part of the contract, but the exploit in effect allows you to make the deputy do your bidding. In the case of Primitive Finance’s code, Uniswap was the trusted deputy and the bug meant that a malicious user could execute a flash swap with outside parameters. According to Yannis, this attack vector is not novel, but it’s not exactly common either.
The Dedaub team, now realizing the implications of the exploit, pinged Mitchell and his co-founder, Duncan Townsend on Saturday to facilitate the disclosure process. Within about 15 minutes, Duncan had confirmed that the vulnerability was, in fact, serious business.
That’s when Alex received the call, and that’s when the clock started ticking.
“Those first few hours for me — I was shocked, and my head was foggy,” Alex told me. “I wasn’t aware of this attack vector at all, but in the disclosure call I knew I had good support.”
You need a war room with synchronous communication to even begin to understand the magnitude of the exploit, how it works, and what can be done to patch it. How do you form a war room in a matter of seconds when you’ve just heard the news and are experiencing shellshock? You pull yourself together and find out who else in the security community is up and has the right skills and temperament to man the battle stations. On Saturday evening, Central European Time, Alex tapped the Dedaub team for their deep technical wizardry, Mitchell for his comms and organization, and Emiliano Bonassi for his leadership and expertise. Alex pulled in Emiliano, a well-known DeFi security researcher and recent founder of ReviewsDAO, as he had seen Emiliano’s name pop up a few times in previous postmortems and knew of his talents.
“We’ve got a real problem. We’re setting up a war room and could use your help,” Alex said.
“How much is exposed?”
“More than a million.”
“I’m in,” Emiliano said.
Part 1: The Gathering
The war room is not an easy place to be in. There’s everything at stake, and the fog of war means you’re flying blind. The dark forest is blurry, hazy, and full of unknown predators looking to end your time on chain and steal user funds. There’s no telling how much time you have to act, or how many chances you have to get it right.
“You have to immediately get your bearings and assess what’s going on,” Emiliano told me. “Usually, I start with questions to figure out who everyone even is and whether the war room is full of trusted players. In this case, we had a lot of advantages: I had Mitchell to work with me on organization and communications, since good communications within the war room and also with users and investors is extremely important.”
As a veteran of many war rooms, Emiliano took charge alongside Mitchell and in rapid fire succession began the strategic planning: designating the players, the timezones, responsibility for various roles, the tasks, and the roadmap ahead.
What made this war room more challenging than the usual fare is that Primitive Finance is a truly decentralized protocol. There are no admin keys, no multisig wallet. The contracts can’t be paused. Once they’re live, they can’t be changed.
After the Dedaub team walked everyone through the proof-of-concept exploit, a potential solution emerged: what if the war room ran a fake airdrop or trading competition to tempt users into resetting their approvals to 0, which would insulate them from a potential hack without users even knowing they were protecting themselves?
But right from the start, there were important objections to this approach.
“We need to make sure we’re not rushing into an approach based on stress,” Yannis said. “And we have to have a reasonable Plan B.”
“If this goes poorly, and people don’t buy that an airdrop is actually happening or not enough users reset, this might reveal the exploit itself,” Neville added. “Plus, transparency with the users is important.”
In retrospect, Plan A maybe didn’t seem like the best plan. But hindsight isn’t worth much. From the outset of the war room, the vulnerability was daunting enough that it seemed like a whitehack could take a week — maybe more — to develop. Given that at least some people knew about the exploit, would Primitive Finance be able to survive a whole week without word leaking out and a blackhat draining users’ wallets? Even if there were no leaks, what if a blackhat independently converged on Dedaub’s findings?
Ultimately, Alex decided that transparency was the best way forward, which decisively ruled out Plan A. He made the right call when it didn’t even look like there was a right call.
So, what was Plan B?
Plan B was a whitehack. If you can’t get approvals reset to 0, and you can’t pause contracts, your only option left is to hack the funds yourself, so that you can personally return them to users.
In the war room, this idea understandably turned up the temperature. By a lot. A whitehack isn’t a trivial operation. It can’t be executed haphazardly, it has to be mistake-free, it needs to be itself well-protected from attacks, and it has to be total, capturing all user funds — or at least close to total. All the addresses have to be found and the exploit carefully handcrafted to do the job. There’s zero room for error.
Back in the war room, it’s now midnight, and the stress level is through the roof. It’s finally settled: it’s time for Plan B.
The strategy is readjusted. Mitchell builds out the comms plan for communicating about the whitehack to Primitive Finance users and the wider DeFi community. Yannis, Neville, and Emiliano start work on developing the whitehack through the night. The clock is ticking.
Fast forward to 5am. Yannis is still awake, working on the whitehack and firing on all cylinders. Rumor has it that Yannis didn’t sleep that night. The excitement made it easy.
“We were exhausted, and this had started on Friday, the night before the disclosure, because this was a complicated hack,” Neville told me. “We were initially unlucky because we picked one of Primitive’s contracts that was actually misconfigured. It was vulnerable but because there was a small technicality, the proof-of-concept exploit didn’t work. At that point, we thought maybe we hadn’t even found a vulnerability at all, and went to bed exhausted. Only the next morning, after more work, we realized that it was definitely there. By nighttime, with the war room in full operation, we were running on fumes.”
Part 2: Plan B Encounters a Devastating Challenge
It’s now early Sunday morning. The Dedaub team has written version 1 of the whitehack.
Having just woken up, Emiliano logs into the video chat.
Everyone in the war room is staring: he’s…not wearing any pants. He’s struggling to put his pants on with one hand, while holding the laptop with the other. The whitehat life is hard to keep up with.
After some laughs, like a pro, Emiliano recovers his composure and starts looking over the code, which is then encapsulated in a test suite. Duncan also began reviewing the night’s work.
“Give me some time with it. I’ll make some changes,” Emiliano said.
Further tests proved successful, but Emiliano came to a horrifying conclusion: while the exploit seems to be a success with respect to the regular tokens, it didn’t actually work for non-conventional tokens, like options tokens or liquidity pool tokens. And that’s when panic set in. The whitehack was supposed to be ready to go for launch shortly. Time felt like it was running out. You could practically visualize a blackhat on the other side of the dark forest mirroring your keystrokes — -and going a little faster than you.
And now, it seemed like the whitehack only half-worked, at best.
“We were in a rush before, and the Dedaub team had worked through the entire night,” Mitchell told me. “It was definitely a low point in the story when we realized we might be screwed. Waiting more days with millions at risk was not an option.”
Any attack on the option or LP tokens would be much more complicated, and there was almost certainly no way it would be ready in time for launch. A couple hours struggling with the options token proved unfruitful, and the LP tokens were even more complex. For there to be an effective whitehack to cover the LP tokens, a whole stock market-like apparatus would have to be simulated, in order to drain stocks and then convert them into dollars.
Nevertheless, Emiliano and the Dedaub team raced to create a whitehack variant to steal the options and/or LP tokens before launch time.
It was now 5pm UTC on Sunday.
“It took far too long to develop a whitehack variant,” Emiliano recollected. “We ended up wasting a lot of time on Sunday because there was no quick way to hack options or LP tokens.”
As is common in war rooms, the timing never works out the way you want it to, and sometimes you chase rabbit holes that lead nowhere. After a full day of delay developing the whitehack variant, the war room reached an important verdict: it wasn’t even clear that the option token was exploitable in the first place. The LP tokens still were, but it would take at least 48 hours more to develop a high-confidence whitehack variant for them.
However, there was a glimmer of hope: if a couple of the high-asset customers — the “whales” — could be identified and cooperated, the funds at risk in terms of LP tokens would be vanishingly small. That would have to do.
Launch time was looming, and at 8pm, the estimate finally came in: at least $1.2 million USD in user funds was at risk of theft at that very moment, with an order of magnitude more depending on user account approvals and LP token allocations. Looking back at Saturday, it’s easy to see why it was at first difficult to make a decision on whether to execute a quick solution or develop a full whitehack. At the start, no one even knew the full extent of the funds at risk.
It was time to proceed with the original whitehack.
Part 3: Pressing the Big Red Whitehack Button
No one is ever fully prepared to whitehack their own protocol. The pressure and anxiety are enormous. Sure, the tests have been run, but when the red button has been pushed, there’s no going back. Whatever happens, happens.
“I was concerned to execute it from my own machine,” Alex told me. “I was prepared to do it, but realized that if something went wrong and we’re right in the middle of doing this, it could get really messy. Someone could see the transaction as it was happening, or a bot could find it in the dark forest.”
In the final moments before the whitehack, when hesitation presented itself, both Emiliano and Mitchell encouraged Alex to go through with the execution.
“Alex, you’ll do just as good a job pushing this button as anyone else. You need to do it, and you must do it. You are the chosen one,” Emiliano said.
Over the next few hours, Alex expertly set up the whitehack on his machine and performed some test transactions on a fork of a mainnet. Everyone was fairly confident the thing was going to work.
“After talking to Mitchell and Emiliano, we were ready to do it,” Alex told me.
Neville then posted the only meme capable of transmitting the spiritual power necessary to carry out this most nail-biting task.
And with that exhortation, at 1:05am UTC, Alex took a deep breath and pushed the button.
Part 4: Saving the Whales and Ending the Story
The story was almost over, but since the timelines had been written, rewritten, thrown out entirely, and rewritten again, the team had forgotten to reach out to the whales right before the whitehack to help save their LP tokens, since the whales represented a large percentage of LP tokens at risk. Luckily, the team already knew the identities of a couple of the biggest whales.
Alex immediately reached out to 0xMaki at SushiSwap.
“It’s okay if it was hacked, I’m having dinner,” 0xMaki replied at first.
Then, after a moment of self-reflection, 0xMaki jumped out of dinner, and called a cab to speed home and execute the instructions.
“Bribe the driver to go faster,” Mitchell said.
Calvin Liu, the second whale and well-known crypto investor, also came into the chat the team had set up and reset his approval to 0.
With those two whales accounted for, within an hour of the whitehack, more than 90% of user funds had been saved, with that figure increasing by the minute as more and more users reset their approvals through the hastily modified frontend, designed by Zachary Thielemann at Primitive Finance.
Within a few more hours, all user funds had been saved.
After Mitchell hit go on executing the full comms plan to let Primitive Finance users and the DeFi world know exactly what had happened and the need for users to reset their approvals to keep their funds safe, news of the successful whitehack spread like fire across social media. Private chats and Telegram groups all over the world started lighting up.
In the war room, there was celebration — and exhaustion.
And social debts to be paid. Yannis’s wife’s birthday was on Saturday. Alex’s dad’s birthday was on Sunday. Emiliano’s girlfriend’s birthday was on Sunday. All were cut short by the demands of the war room, but the war room is a jealous god, requiring sacrifice, but also producing heroes.
In the aftermath, Primitive Finance awarded $10,000 to Emiliano for his invaluable efforts as war room leader, and $25,000 to the Dedaub team for finding the bug with their whitehat wizardry.
But the largest award came from the Founders Bounty, funded by Robert Forster and hosted by Immunefi. The Founders Bounty awards 250,000 $ARMOR tokens to anyone who finds a critical smart contract bug in any Ethereum ecosystem project that puts more than $1m in user funds at risk. At the time of publication, this award is worth $188,000. And it’s going to the Dedaub team. This amount comes to just over 10% of the funds that could have been stolen from Primitive Finance, a satisfying outcome for everyone and a perfect illustration of how bug bounties align incentives to secure the community.
This story had a happy ending. Alex knew that running a war room with a fighting chance of saving a project meant picking the right war council. And Alex picked the right one. War rooms are successful to the extent that they have advisors who embody experience, trust, communication, true talent, professionalism, strategic thinking, disclosure, transparency, and many more attributes than can be listed here.
This story had a happy ending. Not all war rooms do.