Disclosure: Fixing a critical bug in Optimism’s Geth fork

Funds Are Safu

On February 2nd, the Optimism team was alerted by Jay Freeman (saurik of Cydia and Orchid fame) to the existence of a critical bug in Optimism’s Geth fork. The bug made it possible to create ETH on Optimism by repeatedly triggering the SELFDESTRUCT opcode on a contract that held an ETH balance.

Analysis of Optimism’s chain history showed that the bug was not exploited (the bug seems to have been accidentally triggered on one occasion by an Etherscan employee but no usable excess ETH was generated). A fix for the issue was tested and deployed to Optimism’s Kovan and Mainnet networks (including all infrastructure providers) within hours of confirmation. We’d like to thank Infura, QuickNode, and Alchemy for their fast response times.

We also alerted multiple vulnerable Optimism forks and bridge providers to the presence of the issue. These projects have all applied the required fix.

If you are running a replica, please make sure to upgrade to l2geth version 0.5.11 as soon as possible or you may fall out of sync with the rest of the network.

For those interested in more information about how the bug worked and was found, we highly recommend reading saurik’s detailed breakdown. We’re awarding the maximum bounty amount of $2,000,042 as part of our official Immunefi bug bounty program. We’re extremely grateful for hackers like saurik for helping to keep Optimism safe.

Key takeaways

We treat moments like these as a chance for self-reflection. This event highlighted several key areas for potential improvement to Optimism’s internal bug-handling process. In the spirit of transparency and collaboration, we’d like to share some of the most important takeaways with you.

Decentralization begets complexity

Optimism has grown a lot over the past year. Early on in our lifecycle, the release process involved coordination with only a handful of key infrastructure providers. Today, between bridges, more providers, and even multiple mainnet forks of our codebase, it’s a different story. It’s great for decentralization, but it adds complexity to releases. And security releases bring even more complexity — we can’t immediately publish an obvious patch, or we risk someone reverse-engineering the vulnerability before anyone upgrades.

To combat this complexity, we opted to fix this issue using a process similar to Geth’s silent patches. First, we made a private patch which we could share with key parties as quickly as possible. Then, once we were confident the hotfix was deployed, we publicly released the fix hidden in an inconspicuous commit. However, unlike Geth, we’re choosing to disclose relatively quickly because:

• We don’t have lots of commits to l2geth to bury the fix.
• The disclosure coincided with a planned network upgrade, so people were already in the process of upgrading.
• We’re confident that there are no unpatched critical clients.

Luckily, we were able to handle this patch proactively. But — it’s clear that the ecosystem will soon be far too large for this to remain practical. We’ll be updating our disclosure protocol to more closely match Geth’s in the near future.

Every line of code counts

This bug stemmed from modifications to the go-ethereum codebase meant to preserve backwards compatibility with our old “OVM 1.0” system (prior to the EVM Equivalence upgrade which cut thousands of lines of our diff from upstream Geth). Simplicity is core to our design philosophy, and it was unsurprising that the bug stemmed from an OVM difference we hadn’t yet removed.

We’re currently in the process of specifying and building our next major release, Optimism: Bedrock Edition. Bedrock Edition will significantly reduce our (already small) diff from upstream go-ethereum. If you’d like to follow along on that progress, it’s all being done out in the open right here.

Bug bounty programs are a necessity

This week has demonstrated the importance of a strong bug bounty program. At the same time as this disclosure, the Wormhole bridge was hacked for 120,000 ETH (~$300m). The Wormhole project attempted to negotiate for a $10m retroactive bounty that has not been claimed at the time of writing. Wormhole is now working on developing a $3.5m bug bounty program. Other large projects should be looking to develop their bug bounty programs sooner rather than later.

We work hard to maintain clean code. By minimizing our diff from go-ethereum, we make it as easy as possible for whitehats to find, test, and report issues. We’re also committed to maintaining the integrity of our bug bounty program, and we couldn’t be more happy to pay out the $2m+42 maximum bounty amount. If you’re a whitehat, we’d love for you to check out our official Immunefi bug bounty program.

And more

We’ve picked some key takeaways to talk about in this post, but we’ll have a full postmortem with the blow-by-blow soon. Stay tuned! In the meantime, you can find more information about the underlying bug in saurik’s detailed breakdown.

Parting words

Last week, we were able to quickly and successfully roll out a patch for a critical security vulnerability to Optimism and all of its major forks. The Optimism network did not experience any downtime and we were able to confirm that the issue was never exploited. We’ve learned a lot over the past week, and see plenty of room for improvement, but we’re more Optimistic than ever.

We’d like to thank saurik once again for finding this bug and working with us as we implemented the fix. If you’re a fellow whitehat, we’d love for you to go check out our bug bounty program and help keep our protocol secure. Thanks for reading, and stay Optimistic!