Opyn ETH Put Exploit Post Mortem

This post provides information about the Opyn oETH put exploit. Specifically,

• what we are doing about it,
• the timeline of events of the Opyn oETH put exploit,
• an analysis of the exploit itself,
• details about the white hat actions taken to recover at risk funds, and
• next steps going forward.

oETH Put users: Please see this first post for an overview of the exploit that drained funds.

Users of other options: Note that only ETH put oToken contracts were affected.

What are we doing about it?

We have the utmost respect and empathy for our early users and want to do right by our community.

For put sellers: We are reimbursing ETH put sellers in full who were affected by the vulnerability via the process outlined here. All oETH put sellers have now been reimbursed in full.

For put buyers: Further, in order to provide liquidity to oETH put buyers, for the next two weeks, we will buy ETH put options for 20% above Deribit best ask price. Additionally, in the case that any unsold oETH puts end up in the money before expiry, users will be able to exercise by sending us a message on Discord.

For all users: Please do not create any new oETH put vaults or buy/sell oETH puts except through the process defined above with the Opyn team.

We understand that this was an extremely stressful day for users. The exploit itself is on us. We let users down. Reimbursing all put sellers in full is the fair path forward. Our goal now is to regain user trust. We will do this by:

• Rehauling our security practices
• Setting up a program of transaction monitoring
• Setting up a rigorous testing period for every solidity code release
• Putting in pause functionality for contracts we release in order to halt attacks
• Increasing the scope and size of our bug bounty program
• Going through full audits for every single piece of code we release
• Increasing our emphasis on test driven development and making security a cornerstone of Opyn.

We will do what we can to make this right. We will come back stronger.

Timeline of Events

9:59 PM PT — a user asks about a suspicious transaction in our general Discord

3:05 AM PT — a user posts that they have trouble withdrawing collateral

4:00 AM PT — we started looking into the transaction to determine if it is problematic

5:20 AM PT — we assemble the team, after determining that it is either a large bug or a hack

5:40 AM PT — we contact trusted security and incident response advisors, incl. samczsun, Taylor Monahan, OpenZeppelin, and Jared Flatow

6:00 AM PT — we remove all of the liquidity from oETH Puts (and preemptively from other options too), so users cannot interact with the affected contracts and so the attacker cannot get the oETH liquidity needed to execute potential further attacks. We also update our frontend to block off interactions with oETH Puts.

6:30 AM PT — we identified the exploitable line of code, stemming from this line. We raised the exercise fee to the maximum of 10% to reduce the profitability of an attack, and allow us to recoup a percentage of funds if anyone was to exploit this vulnerability again

7:00 AM PT — The White Hack Group, Bokky Poobah, and Harry Denley provide additional input, validating our analysis of the issue

7:30 AM PT — Having identified that additional user funds were at risk, we limit our public discussion of the exploit since other oToken holders who knew about the exploit and held oTokens could recreate the exploit and steal the remaining funds. We begin a detailed analysis of how many funds are still at risk, and from which users and vaults.

8:00 AM PT — We tally and confirm that 572,165.13 USDC in the put contracts was still at risk. We start analyzing an additional attack vector where an attacker mints oTokens instead of buying oTokens to understand how profitable the attack would be, if it had already occured, and to determine a safe whitehat hack strategy.

9:00 AM PT — We design a method to safely remove a significant portion of the remaining USDC collateral from vaults. In the absence of a pause function, with the help of samczsun and Jared, we begin to whitehack the remaining funds. The fastest and safest way to do this, we find, is to write a contract that can atomically increase the collateralization requirement and then liquidate all users and then lower the collateralization requirement.

9:30 AM PT — Users begin to discuss the suspicious transactions as an exploit. We begin to draft a statement on the exploit trying to balance the need to inform users with the risk that too much disclosure could put additional funds at risk.

10:00 AM PT — We release a public statement on the exploit, informing users but not revealing the details of the exploit until we had removed all of the at risk USDC collateral safely.

11:40 AM PT — samczsun helped us in successfully executing the first white hack

2:30 PM PT — we finished rescuing the first 439,170 USDC from the contracts. We started brainstorming ways to rescue the remaining funds, some of which were in the attacker’s vaults.

3:30 PM PT — we release a public statement informing the community about the first white hack, an overview of the exploit, and initial actions for affected ETH put users.

4:00 PM PT — we determine another way to to execute a second white hack to save all of the remaining funds

6:30 PM PT — samczsun helped us recover 132,995 USDC through the second white hack. We updated the community about the second white hack.

10:45 PM PT — we updated the community about reimbursement

Aug 5, 9:55 PM PT — we updated reimbursement for ETH put buyers to include the choice to exercise their option with Opyn if it ends up in the money before expiry

Aug 6, 10:59AM PT — we updated ETH put sellers about the reimbursement process

Aug 7, 10:50AM PT — we posted a list of affected ETH put seller addresses and balances for ETH put sellers to review

Aug 9, 10:00AM PT — we reimbursed all ETH put sellers in full

The Exploit

• Total amount of funds at risk: 943,425.13 USDC
• Total amount lost in hack: 371,260 USDC
• Total amount safely recovered via whitehacks: 572,165.13 USDC

Background on the msg global variable in solidity:

The way that ‘msg’ global variables work in solidity is that the `msg` object exposes information about the current contract call.

Consider the following example with a simple contract called Bank which has a depositCheque function. The external depositCheque function calls an internal helper function _depositCheque.

Example:

Background on Opyn and Options:

Opyn options give the option holder the right but not the obligation to sell or buy their underlying asset at a pre-specified price called the strike price. The option holder can exercise their option and sell or buy the underlying assets at any time before expiry of the option.

In this specific case, the options at risk were the oETH put options which gave the option holder the right but not the obligation to sell their ETH for some pre-specified amount of USDC anytime before the expiry of the option.

The Exploit:

The exercise function that Opyn uses has an external function that then calls an internal function. The internal exercise function is called in a for loop by the external function.

The issue arises on line 809 which checks that the ‘msg.value’ for the exercise internal function matches the underlyingToPay. Since the internal function derives the msg data from the external function, calling the internal function multiple times passes the same msg data multiple times to the internal function.

For ETH put options, the exercise function does not work for exercising from multiple vaults at the same time, other than the exploit described below. The problem arises when the attacker calls the external function oToken. exercise (oTokensToExercise, [vaultAttacker, vaultVictim]) on a vaults where the states are as follows:

vaultAttacker

• oTokensIssued: oTokensToExercise/2
• Collateral: X USDC

vaultVictim

• oTokensIssued: Some Amount >= oTokensToExercise /2
• Collateral: Some Amount >= X USDC

The internal _exercise(…) function will require that the msg.value == oTokensToExercise i.e. the amount of ETH passed in is equal to the number of options exercised. However, calling the external exercise function with 2 vaults can trick the internal _exercise function into thinking 2 ETH is sent exercising 2 options, when really only 1 ETH is sent. Thus, the attacker, without paying underlying tokens for the exercise on the victim’s vault, takes out X USDC as collateral.

The exploit requires oTokens to execute. The oTokens could either have been bought on uniswap or minted by the attacker. In the case where oTokens are purchased on uniswap, the net profit of the attacker = X USDC - cost of acquiring the tokens, absent any exercise and gas fees. In the case where oTokens are minted, the net profit of the attacker depends if their vault is later attacked by someone else. In this case, the attacker’s worst case net profit is breaking even and the best case is X USDC, absent any exercise and gas fees. The second attack vector is unlikely to be profitable before expiry (as the attacker’s left over vault could be attacked by others), but in the final block before expiry, the attack could steal all of the collateral in all vaults.

The Attack:

Given the above vulnerability, we now know that the attacker can pay the underlying one time, and use that for multiple exercises within the same `exercise` call, as long the exercised amount on each vault is the same size.

Looking at one of the attack transactions we can see how the attack was conducted. The attacker wrote a wrapper contract which conducts 4 actions within 1 transaction:

1. Create vault
2. Mint 30 oTokens
3. Call exercise, specifying the victim’s vault and their own vault to be exercised on.
4. Redeem the underlying in their own vault.

Do notice that the attackers have to obtain 30 oTokens before the attack. In the case above, we noticed that the attacker bought the oTokens on Uniswap just prior to the attack.

Does this attack work on non ETH Puts?

This attack doesn’t work for non ETH puts. If the underlying is an ERC20 token, then msg.value is not used. Instead, the underlying ERC20 token gets transferred into the contract to the right vault on each internal function call.

The way ERC20s are transferred is different from how ETH is transferred. For ERC20s, the smart contract pulls the required amount of ERC20 from users into the smart contract itself on each internal exercise function call. On the other hand, ETH is pushed by the user (msg.value) to the contract on the first external function call, and each consequent internal function call merely checks that the pushed ETH amount matches the amount needed to exercise.

Response

Couldn’t Opyn have turned off once the exploit was discovered?

In short, we can’t turn the protocol off. Opyn is permissionless and decentralized by design, and Opyn contracts are not able to be turned off or disabled. We took action as aggressively as possible to minimize further damage once the exploit was discovered. This included buying additional Put oTokens to prevent further attacks, removing the ability for Put oTokens to be sold, as well as white hacks on existing Put sellers to ensure that their collateral was safe.

Increase Exercise Fee

We realized that since the attacker was exploiting a vulnerability in the exercise function, if we raised the exercise fee the protocol receives on each exercise, we could recover some of the funds as a fee. We increased the exercise fee to 10%, the maximum amount the fee could be set at. Increasing the exercise fee was effective at reducing the profitability for the attacker from both the attacks mentioned above. Removing all collateral was the only way to protect users’ funds.

Pull Liquidity and Buy Put oTokens

Initially, we recognized that an attacker needed to possess oTokens to execute the attack and that anyone who had oTokens could replicate the attack if they understood the exploit. To mitigate further losses and attacks, we removed liquidity from our ETH Put pools on Uniswap to prevent attackers from buying these oTokens. We also removed the ability to buy ETH Puts on the opyn.co website. To ensure liquidity for existing oToken holders (and reduce the possibility of long token holders attacking the protocol while user funds were locked in), we offered and continue to offer to purchase all ETH Put oTokens that were outstanding at the time of the exploit for 20% above best ask price on Deribit.

Whitehack #1

The goal of the first whitehack was to rescue as much of the user funds as quickly as possible. Responding quickly was the most important aspect for the first whitehack.

At a high level, the rationale of the first whitehat hack was to use our limited admin privileges to remove money from vaults. One way to do this, we realized, was to liquidate all vaults with USDC collateral in them.

The first whitehack worked as follows in an atomic transaction:

1. The admin increases the minimum collateralization requirement for all vaults to infinitely high and sets the liquidation factor to 100%. We also set the liquidation incentive to 20%. This causes all vaults to be undercollateralized, fully liquidatable, and pays the liquidator a 20% liquidation incentive.
2. Using all the oTokens that we had taken out from the liquidity pools, we liquidated as many vaults as we could. We needed only about 80% of the oTokens that a vault had issued to be able to fully liquidate that vault because of the 20% liquidation incentive.
3. The admin decreased the minimum collateralization ratio back to 100% at the end of the process and set all the liquidation parameters back to 0.

The issue with this strategy was that we needed oTokens to be able to liquidate vaults and we didn’t have all the oTokens in circulation. We needed at least 80% of each oToken in circulation to fully be able to liquidate the corresponding contract. We had most of the oTokens for the ETH $330 Put and ETH $200 Put, but significantly less for the ETH $180 Put. For the ETH $270 Put, the attacker owned about 40% of all oTokens in circulation, which meant we couldn’t liquidate and remove a significant amount of USDC collateral. Hence, we started trying to buy back oTokens so that we could liquidate more vaults.

Whitehack #2

We already secured most of the user funds with the first white hack. The goal of the second whitehack was to remove and secure all the remaining funds in the contracts. The second whitehack worked the exact same way as the first whitehack, except for a key difference in step 1. Because we did not have any additional oTokens, the admin took out a flash loan to mint a large number of oTokens. Steps 2 and 3 were the same, with one important note being that we were able to liquidate the admin vault created in step 1 by having at least 80% of the originally minted oTokens by the end of the liquidations of user vaults.

This process worked to fully rescue the funds from the contract. The only funds we were not able to drain were the 1080 USDC from the attacker’s vault and the 151.162 USDC from vaults which had not issued any oTokens since they could not be liquidated. The owners of the vaults that had not issued any oTokens can still redeem their collateral normally through the protocol.

Why wasn’t the issue with the exercise function caught at audit?

We had implemented a proportional exercise function that we sent to audit. This worked by proportionally removing collateral from the total supply of collateral from all options sellers and paid out the buyer upon exercise. While the code was at audit, in testing, we realized that there was an issue with the way that proportional exercise was implemented.

The issue with the way we had implemented exercise was that it forced all options sellers to have to wait till expiry to be able to withdraw underlying. Forcing options sellers to wait till expiry to redeem underlying was strictly worse for sellers since they could lose a lot more money if they had to wait till expiry to redeem and sell their underlying, since the underlying could crash further by the time expiry showed up. This was a core financial issue.

We reported this issue to Open Zeppelin as a bug, changed the code to reflect a fix, and re-sent the new commit to Open Zeppelin.

Having been sent after the audit had already begun, this new commit was not in the final version of the code that was being audited. This was a failure of communication on our part, as we did not clearly communicate to Open Zeppelin that this should be in the scope of the review process. In the future, it is on us to ensure 100% testing and audit coverage for every single line of solidity code on the contracts.

What will Opyn do in the future to prevent this from happening?

The security of the Opyn protocol has always been and continues to be our highest priority. We have let our users down and will work tirelessly to rebuild your trust. We are taking the following steps:

Internal Testing and Review

• The entire team, including those who work on code other than smart contracts, will review the spec, and undergo a period of trying to attack it before any code is written to find any mechanism related issues as soon as possible
• Every single line of solidity code will be written and reviewed by at least 3 Opyn developers and upon completion of that, every line will be independently reviewed by external auditors
• Define and test all high level workflows and common error flows for each systems
• Build a stronger testing process, by defining all critical testing scenarios before any tests are written
• Have an internal breakathon where the team tries to break our own codebase before we release anything to mainnet
• Verify system invariants with Trail of Bit’s Echidna system

Pause

• Introduce pause / emergency shutdown functionality to ensure we can turn off the system in the case of something suspicious

Audits

• Release fully audited code and not make any changes post audit without review
• Continue to work with top auditing firms such as OpenZeppelin and Trail of Bits

Bug Bounty

• Increase bug bounty rewards for our existing Bug Bounty Program by 50% to $15k for high vulnerabilities and $60k for critical vulnerabilities

Responsiveness

• Improve our responsiveness to bug reports
• Ensure that all our users and everyone in the community knows that they can reach us promptly if they email security@opyn.co.
• Set up an urgent channel in our discord which users can also use to reach us quickly.
• Set up a proper system of alerts from the security email and the urgent discord channel.

Monitoring

• Set up 24/7 monitoring practices to quickly detect suspicious transactions that could point to potential vulnerabilities
• Set up dashboards to help us easily know how much money is in the system at every moment. We should easily be able to access the snapshot of our system at every second in the history of its existence.

Security Advisors

• Engage a set of security advisors who will help us consistently improve our security processes

When can we expect the ETH puts to be live again?

Currently, we are considering launching WETH puts at some point in the coming two weeks, since WETH is an ERC20 token unlike ETH. We are currently fixing the vulnerability in the v1 ETH puts, adding admin pause functionality, and going through a re-audit of any code that was outside the bounds of the Open Zeppelin audit. ETH puts will only be live after the audit process. We will keep the community updated as we have more details on the timeline for ETH puts.

DeFi Community

We really appreciate the DeFi community’s support and everyone who has reached out as we’ve been navigating this incident. Special thanks to samczsun, Tom Schmidt, Jared Flatow, Taylor Monahan, Alejo Salles (and the rest of the Open Zeppelin Team), Josselin Feist, Haseeb Qureshi, Andres Bachfischer, Martin Abbatemarco, Geoff Hayes, Reuben Bramanathan, Robert Leshner, Bokky Poobah, Harry Denley, The White Hack Group, Peckshield, Tina Zhen, Sunny Aggarwal, Kevin Britz.

Please let us know if you have any additional feedback or questions. Our priority is to be there for our users. You can reach us on Discord and feel free to DM us.