Postmortem: The March 9th Hack on Hedera

Between March 9–11, 2023, around ~$600k was exploited from the Hedera ecosystem and an ecosystem-wide collaboration led to the pausing, remediation, and replenishment of all user funds.

Identifying the hack

On March 9th at around 4:30 GMT, we were alerted to an issue via a support ticket in the Pangolin Discord. Our engineering team immediately began investigating, and within two hours, we confirmed malicious activity in our contracts.

At that time, we identified that approximately $20,000 had been taken from Pangolin and about $2,000 from Heliswap. A war room was created in collaboration with the Heliswap, the HBAR Foundation, and Hashport teams to further investigate the issue and confirm the nature of the hack.

The situation escalated rapidly as they exploited hundreds of thousands more from Pangolin’s farm contracts in the USDC/USDT and USDC/WHBAR pools.

Addressing the hack

The hacker used the Hashport bridge to send tokens to Ethereum, with their last transaction occurring at 10:30 GMT. Their Ethereum address can be traced here. We quickly reached out to Hashport to pause the funds, and they responded swiftly by pausing both the bridge and their HTS tokens. Various parties, including Hedera, Stader, and SaucerSwap, were alerted to the issue.

The hacker later attempted to transfer their funds to Binance, ChangeNow.io and Godex.io, the CEXs were contacted to halt the activity, and authorities were alerted to the situation.

After analysis, the Hedera DevOps team shut off proxy access to the Hedera mainnet at 20:18 UTC March 9th. This prevented users from accessing the mainnet (and the attacker from draining additional tokens), but the mainnet remained up.

The proxy access was shut off because it was discovered that any contract managing HTS tokens allowed a user to call an arbitrary address from the contract that would enable its HTS token managed by anyone else. More details from Pangolin Engineer, Shung, here.

In the following day, a code change was authored for Hedera, preventing a smart contract from using a delegate call to call a precompiled contract. You can find Hedera’s detailed analysis here.

Within hours of the network proxy access being turned on, all USDC/USDT and USDC/WHBAR user tokens were fully restored to user accounts.

Pangolin replenished affected user funds with the support of the HBAR Foundation.

The swift action taken by our Pangolin’s Moderators, Engineers, and the Hedera community prevented the potential draining of approximately $30 million in Hedera DeFi TVL.

A huge amount of thanks goes out to everyone at Pangolin, Heliswap, Hashport, The HBAR Foundation, and SwirldsLabs for helping this situation, but a special tip of the hat goes to the Pangolin DAO contributors for their foresight and swift action.

Takeaways

The actions and systems at Pangolin allowed us to quickly identify, address, and communicate the exploit and vulnerability. To further increase our awareness of potential vulnerabilities we’ve strengthened communication lines about bugs/issues between our moderators and engineers.

Everyone’s unwavering teamwork and tireless drive to create a solution was nothing short of remarkable. Turning around a network level vulnerability in 48hrs and replenishing all user funds is a testament to the resilience of this community and an unwavering commitment to protecting the community’s interests.

For further questions on the events, join the Pangolin Discord here.