Sorbet Finance Vulnerability Post Mortem
How the Gelato team and honorable community members rescued $27M at risk from an attacker.
On December 11th around 10:15 CET, we were alerted of a critical vulnerability in a smart contract used on Sorbet Finance. This contract was designed to enable users to enter and exit G-UNI pools from Sorbet Finance with just a single token rather than being required to provide an equal proportion of two from the get-go.
Around 22:00 CET, the Gelato team successfully executed a whitehat hack to rescue vulnerable users’ funds, totaling about $26M. These funds were sent to a secure escrow contract from where only the original owner of those funds could reclaim them, providing that they first followed the necessary steps to ensure all outstanding approvals to the vulnerable contract were revoked.
No other smart contracts or components of Sorbet Finance, G-UNI, or Gelato Network were affected. G-UNI pools in particular were safe and continue to be safe. Wallets that did not interact with this vulnerable smart contract via Sorbet Finance are not affected. If you believe your wallet might have been affected, please use this tool to check and then revoke all approvals via the Sorbet UI. If you are not sure how to proceed, you can contact us via our Telegram channel and our team will help you.
On Friday, December 17th, after going through an extensive review and auditing process, the Gelato team deployed a new smart contract which made interacting with pools via the Sorbet UI safe again.
A critical risk remains for any user who has yet to revoke approvals to the smart contract at risk. Therefore we strongly urge users who have interacted with the Sorbet Finance pools page to revoke their approvals first before sending any tokens to their wallets.
As of December 20th at 12:00 CET, all but 66 addresses have successfully revoked their allowances. We continue to actively monitor and try to rescue affected wallets to the best of our abilities but for the purpose of ensuring the ultimate safety of their funds, affected users have to revoke their approvals on Sorbet themselves.
Timeline of Events
Note: All times listed are in CET
December 11, 2021
- 10:16: The Gelato founders were alerted of a critical vulnerability by Yash Shah (who discovered the bug) and Samczsun.
- 10:18: A War Room was created to discuss the bug and the process to secure the funds at risk. Shortly thereafter, the Gelato team confirmed the vulnerability.
- 11:26: The Gelato team decided to conduct a whitehat hack to rescue users affected by this vulnerability.
- 11:45: The Gelato frontend team started developing features that enable users to revoke approvals on Sorbet Finance.
- 22:00: The initial whitehat hack which secured funds worth $26M was executed via a private transaction using Flashbots. Additionally, the team started running whitehat bots that monitor the mempool for any incoming transactions which would put users’ funds at risk, and conduct additional whitehat hacks to secure funds.
December 12, 2021
- 12:38: The Approval Revoke UI went live, the Gelato team made announcements on all social channels to urge affected users to revoke their approvals on Sorbet Finance immediately. The team also reached out to various Web3 projects and crypto influencers and asked them to retweet for maximal information dissemination.
- 19:24: Prominent whitehat Searchers and Flashbots community members 0x911, imagine, and Bert Miller, spun up an additional bot and joined the battle against the malicious blackhat hacker who was trying to exploit vulnerable users funds.
December 13, 2021
- 16:54: We reached out to all prominent centralized exchanges which the affected wallets interacted with and urged them to notify the owners of the accounts at risk about the vulnerability and freeze those accounts to protect the funds. Moreover, the malicious black hat’s account was frozen and an investigation was started to take legal action against the account holder.
- 17:21: Etherscan Alerts and banners went live, showing users if their wallets were affected and the steps to follow to revoke their approvals.
19:45: The Gelato team sent out over 250 alert transactions to users at risk which showed up on top of their Etherscan transaction overview. The sender of these transactions had detailed information on how to secure their funds displayed on its Etherscan page.
The team also worked tirelessly to try and reach out to affected users via all available social channels by tracking down their identities via transactions that were linked to their ENS names. The goal was clear, get users to revoke approvals NOW.
December 18, 2021
14:30: Users that were still at risk received an NFT airdrop alerting them about the vulnerability and urging them to remove approvals.
The Gelato team successfully executed 35 additional whitehat rescue transactions that secured another $1M from wallets at risk. All but 66 users have yet to revoke their approvals on Sorbet.
The smart contract containing the vulnerability had a similar bug to the one that was revealed in the dYdX vulnerability report posted on December 10th. Thanks to this report and our informant, we were able to discover the vulnerability on Sorbet and rectify it before a blackhat hacker could exploit it.
The core functionality of the vulnerable smart contract was to enable users to enter G-UNI pools with only one token, rather than being required to provide an equal proportion of two from the get-go. This required the smart contract to conduct a swap on a DEX aggregator before depositing the funds in the pool. This swap was done on 1inch, which requires other smart contracts to pass certain addresses and calldata generated by the 1inch API to execute the right functions on the 1inch smart contract.
Enabling the smart contract at risk to conduct arbitrary low-level calls that were intended to only execute swaps on 1inch made it possible for the potential exploit to occur. For any developer reading this, never enable a smart contract to take arbitrary addresses and call data arguments if a) everyone can call this function, b) you don’t do sufficient checks on what the addresses and calldata that were passed should be, and c) users provide this smart contract with approvals.
We urge all dapp developers reading this to double-check their smart contracts in case they interact with DEX aggregators on a smart contract level to make sure their systems do not contain a similar vulnerability to ours.
How Did the Bug Escape Notice?
The Gelato development team was very much aware of the dangers of low-level calls such as `.call()` in Solidity. This bug should not have made it into production and the fact that it did, regrettably shows a serious mistake in our smart contract testing and review process for this particular feature launch. This was the first security vulnerability that made it into production in the 2.5 year existence of Gelato and we will make sure that it will remain the last by upholding and even hardening our usual high standards for all future smart contract releases.
The smart contract at risk only contained a small change compared to the previous audited and battle-tested version. This led our smart contract development team to release this into production prematurely, without the usual rigorous security review process that we typically follow.
This incident is very much preventable and we have already taken several steps to make sure that a bug like that will never make it into production again.
What Have We Done to Prevent This From Happening Again?
We hold ourselves to the highest standards of security in Web3. Through this incident, we have learned that no matter how small a change, and how much battle-tested history a contract has, it must always be thoroughly reviewed and audited before going live in production.
To avoid vulnerabilities like this from happening again, we plan to implement the following security measures:
- A thorough review of all new contracts or all modifications to those contracts by multiple team members, however small the modification is.
- Closely monitor risky patterns and previously reported smart contract exploits like the dYdX report, which in this case, explained exactly our vulnerability.
- Assess the worst-case scenarios of a new deployment or contract modification and design countermeasures.
- Create a bug bounty to incentivize external developers to find errors in the code before we release it to the public.
- Conduct continuous code reviews of smart contracts that have already been in production without any incident for several months.
- Enlist the protocols that we integrate to review our integration of them for security before putting a new feature into production.
Bug Bounty Payment
We are very grateful for the person who brought the vulnerability to our attention quickly and privately. This kind of responsible disclosure allowed us to act swiftly and rescue $26M from affected users.
Moreover, there were a lot of amazing Gelato community members that jumped in and helped our hands on or by connecting to the right people which enabled us to conduct additional whitehat rescues and reach out to as many affected users as possible! We are extremely grateful for your contribution and for helping us in such difficult times.
The Gelato team will put up a governance proposal to give out generous bug bounties to those community members that helped us along the way.
Reimbursement of Stolen Funds
We were successful in rescuing all the vulnerable funds at the point of the whitehat hack; however, users’ funds remain vulnerable until they revoke the approvals on Sorbet Finance. Despite our best efforts and having multiple recovery bots running continuously to save those at-risk funds, a total of around $744k was stolen by malicious searchers which were able to outrace our bots for a small number of transactions at the very beginning. Since then we have optimized our system and outcompeted the malicious searchers for most transactions.
A governance proposal to reimburse these losses in full will be created and shall be voted upon by GEL holders. Due to the likely scenario of faking funds being stolen, the amount of time that has passed since the incident, and the team having gone to extraordinary lengths to try and notify the addresses at risk, we will no longer reimburse any losses that happen after December 20th, 13:00 CET.
Nevertheless, we will continue to operate multiple whitehat bots that will continue trying to save funds that become risky. Funds that will get successfully rescued after the cutoff date, will be refunded to users minus the miner fee incurred by the bot operators in getting the rescue transactions mined.
Despite the vulnerability and failure to adhere to the highest security standards that we aspire to, we managed to conduct multiple successful whitehat hacks and rescued over 27M worth of users’ funds. The Gelato team has dealt with the vulnerability swiftly and diligently, working non-stop to have recovery bots and the approval revoke UI up as soon as possible, in the meantime, communicating on all channels to our users about the urgency of revoking approvals.
We appreciate the overwhelming support from our community. We thank you for your understanding and patience during this trying time. We’ve come out of this incident safer and stronger, and we will continue to push boundaries to bring the best Web3 automation tools to the Web3 ecosystem.
Big thanks to Yash Shah, Samczsun, our caring investors at Dragonfly, Robert Miller, Imagine, 0x911, Nathan Worsley, the Binance and Etherscan teams, Kaito San, and Hory San for their continuous support through this difficult time!
About Gelato Network
Gelato Network is Web3’s premier automation network, enabling developers to automate a wide variety of arbitrary smart contract executions on and across all EVM-based compatible blockchains such as Ethereum. Examples of use cases developers have built on top of Gelato include Limit Orders on AMMs like Uniswap, automatic compounding of yield farming vaults, Aave liquidation protection, MakerDAO debt ceiling updates, automated liquidity management, and even the petting of Aavegotchis.
Our ultimate goal is to automate everything and by giving developers the reliable tools they need to do this, we can empower their users to get the most out of their Web3 experience.
► Check out what we’ve been working on at 🍦 https://gelato.network/🍦
►Automate your smart contract at https://app.gelato.network/
Connect with us: