Multichain Contract Vulnerability Post Mortem

On January 10, 2022, we were alerted to two critical vulnerabilities with the Multichain liquidity pool contract and router contract by Dedaub, which later confirmed to affect eight tokens (WETH, WBNB, MATIC, AVAX, MFI, WSPP, TLOS, IOTEX). The vulnerability of the liquidity pool was fixed soon after it was reported, as we upgraded the affected tokens liquidity to new contracts. However, the risk remains for the users who have yet to revoke approvals for the affected router contracts. Importantly, users themselves have to be the ones to revoke the approvals. As such, we made an official announcement about this vulnerability on January 18 and urged the users to take actions immediately as instructed.

Status Now (as of February 18 24:00 UTC)

• A total of 7,962 user addresses are affected, 4861 addresses have revoked their approvals, while the remaining 3101 addresses have as yet not done so and still need to take action as soon as possible.
• A total of 1,889.6612 WETH and 833.4191 AVAX has been exploited, from which 912.7984 WETH and 125 AVAX were rescued by the joint efforts of Multichain and whitehats.

Daily attack sum, by @Dune Analytics

After one-month efforts to notify all affected users, over 61% have revoked their approvals so far. According to the monitoring data by Dune Analytics, the attacks mainly happened in the first week after the exploitation was released. The hack transaction and amount have plunged since January 25. The past two weeks have seen a few attacks of very low amounts.

Compensation Plan

Together with the joint efforts of whitehat hackers, we have successfully rescued 912.7984 WETH and 125 AVAX from hackers, or nearly 50% of the total stolen funds. However, in spite of our best efforts, a total of 976.8628 WETH has been stolen.

The team initiated a proposal to reimburse 100% of users’ losses, and the funds (including the miner fee) will be returned to users who a) have revoked their approval and b) have submitted a ticket at our help desk. It’s been one month since we disclosed the hack and urged users to revoke the approvals, and the team has pursued every option available to notify all users of the risk. As such we will not reimburse any losses that happen after February 18 24:00 UTC.

We will of course continue to make every effort to rescue as much funds as possible from further exploits, and will keep everyone updated. Any funds rescued after February 18 24:00 UTC will be refunded to users (minus the miner fee).

Again, we strongly urged users who ever gave approvals to the affected token contracts to revoke before sending any tokens to their wallets. Please use this tool to check and revoke via Multichain UI. If you are not sure how to proceed, please follow the steps of our instructions. You can submit a ticket at our help desk if you run into any problems, or contact us via telegram, our team will help you.

Bug Bounty Payment

We are grateful to Dedaub for reaching out to us at the very moment they found the vulnerability, and assisting us to battle against the attacks. The team will reward Dedaub with our maximum bug bounty of $1M for each of the two vulnerability disclosures. We really appreciate what Dedaub has done for the security of Multichain and we have no doubt that they have contributed to a sustainable crypto ecosystem. We‘ll continue to give out generous rewards to encourage vulnerability research and discloure.

Shout-outs to

Everyone who jumped in and reached out to offer help at the critical moment. Big thanks to Etherscan Team, Gelato, Ava Labs, Sushiswap, Spookyswap, Metamask, Opensea, Looksrare, Tether, Popsicle Finance, Frax Finance, Gemini, Synapse Protocol, BlockSec, 0xlosha, MevRefund and all the community members.

Timeline of the Event

Jan 10: A War Room was created shortly after two vulnerabilities reported by Dedaub to discuss the bug. We took a series of actions to secure the funds at risk.

1. We went over all contracts, involving 1500+ supported tokens and narrowed down the affected contracts to 6 tokens (WETH, WBNB, MATIC, AVAX, MFI, WSPP). The vulnerability with the liquidity pools involved with these 6 tokens was fixed within 24 hours.
2. The router contracts with these six tokens were paused.
3. An overall check was conducted with different partners to scan for affected addresses.
4. A real-time monitoring system was set up.
5. A revoke portal on the frontend was established.
6. A secure escrow contract was created.
7. Users were notified to revoke approvals.

Jan 18: We posted an alert announcement to urge affected users to revoke. Meanwhile, Multichain took prompt measures to maximize the proportion of users revoking approvals by continuously posting updates to all users and by reaching out to all possible channels (Opensea, MetaMask, Polygon Bridge, Dapps, etc.) to create alerts and to call for users to act immediately.

16 hours after the alert announcement was released, the first exploit happened. Multichain and Security company Dedaub immediately joined the battle against black hackers by running whitehat bots to rescue users.

Jan 19: To prevent users from being affected, we started to send over on-chain alert transactions to all affected AVAX, MATIC, WBNB addresses.

In addition, we reached out to Etherscan explorer to set up alert banners for exploiters and affected WETH addresses.

Jan 20: After negotiation, one hacker agreed to return 259+63 ETH.

Jan 22: Security company BlockSec joined the fight by conducting whitehat hacks.

Jan 24: We developed an approval-revoke API for Dapps to integrate, by which users can directly revoke approval(s). Dapps including SpookySwap, SushiSwap, SpiritSwap, AVAX bridge, AAVE, etc. have integrated this.

Jan 25:

1. A community member (twitter handle @0xlosha) joined the whitehat rescue and saved 125 AVAX.
2. The vulnerability was found to involve two additional tokens (TLOS, IOTEX) by Multichain. Prompt attempts were taken to solve the bug, which was successfully fixed within 24 hours with no loss.

Jan 29: We assisted WSPP holders with the exploit happened on Jan26.

Feb 14: Tether froze one hacker’s Ethereum address with over $715,000 worth of USDT, thanks to the help of community members.

Feb 17: All affected tokens upgraded to V6 contract, support native coins and don’t need to approve tokens.

Together with the Multichain community, we will continue to do the best we can to trace hackers and protect users funds safety.

Technique Explanation

After receiving the alert from the security company, we did an immediate check and were able to reproduce and verify the vulnerability. We thoroughly checked all the contracts that may be involved. The team finally confirmed that the vulnerability involved 2 contracts, AnyswapERC20 (V5 and Previous versions) and AnyswapRouter (V5 and previous versions). The vulnerable functions are as follows, and they only affect the above eight tokens.

AnyswapERC20:

• depositWithPermit

AnyswapRouter:

• anySwapOutUnderlyingWithPermit
• anySwapOutExactTokensForTokensUnderlyingWithPermit
• anySwapOutExactTokensForNativeUnderlyingWithPermit

Root cause: the vulnerability is generated under the joint action of the Anyswap contract and the underlying token contract. For some underlying token contracts, they don’t implement a permit method, while containing a fallback (non-reverting) function. When its permit method is called, the contract runs without reverting. The consequence of the following operations being executed puts the funds at risk.

AnyswapERC20 contract is used for the liquidity pool. The time the vulnerability warning was received, the team immediately fixed it and deployed the upgraded V6 contract. In parallel, the team sent an alert request to the MPC network to transfer the liquidity at risk into the upgraded liquidity pool. The team then verified that the pool assets were safe.

The AnyswapRouter contract is for asset routing across chains. This vulnerability affects the assets that are given approval to the router contract by users, which means the users are the only ones capable of revoking their own approvals. The team immediately developed a tool launched at the official website to enable users to revoke approvals while we notified users as much as possible in all possible ways to revoke their approvals. In addition, we developed a secure escrow contract and monitored the assets at risk in order to carry out our own whitehat hack rescue of them.

Here is an attack example to explain this vulnerability. The final result was that the assets user gave approvals to the AnyswapRouter contract were transferred to the attack contract.

How do hackers attack?

The hackers deploy the attack contract and set the affected token contract address as its underlying token parameter. Then they can steal the user funds by making calls of anySwapOutUnderlyingWithPermit to AnyswapRouter contract.

• “from” can be the user address that the user approved AnyswapRouter,
• “token” is the attack contract address,
• “amount” is the balance in the user address, and other parameters with arbitrary values.

Actions taken to prevent this from happening again

• Further rounds of security audit. Further rounds of security audits on contracts, cross-chain bridges and MPC will be conducted. The team will make continuous efforts to make security enhancements on the whole cross chain bridge architecture and closely monitor all new contracts.
• MULTI Security Fund. Multichain will initiate a governance proposal of the Security Fund. The security fund is used to take necessary and possible rescue measures for asset losses caused by possible vulnerabilities in Multichain’s own system and service. The establishment and use of the fund will be announced at a later date.
• Bug Bounty Program. We encourage the community to continue to review our code and security. We will work with Immunefi on the Bug Bounty program. This program is set to recognize the value of independent security researchers and teams. We believe it’s important to make the good guys stay motivated and make sure they know they are appreciated. The team will provide a reward of $500 to $1,000,000 for discovering and submitting vulnerabilities. Click here for more details.
• Free public supervision API. The approval-revoke API we developed for this incident has been demonstrated to be effective. The protocols and applications which integrate this API can easily detect and then alert affected user addresses to take actions accordingly. We are updating it and will offer a free public API for all projects.

Thank you all for your patience to learn and understand this incident. We appreciate every supporter and honor your trust in us. We will learn from this incident and emerge stronger and better. We’ve been working hard and will continue to do our best to serve as the ultimate cross-chain router for Web3.