DeFi Security: lessons learned from the $600M Ronin hack

Babylon is a community-led asset management protocol that enables users to create investment clubs (we call them gardens) and invest in DeFi together. It’s built on the Ethereum network and it’s non-custodial, transparent, permission-less, and governed by the community. BABL is the governance token behind it.

Security intelligence series from Babylon

In this post we provide an initial security analysis and some security recommendations based on current knowns but also unknowns of Ronin’s $600M hack recently attributed to Lazarus by the FBI, which is still under investigation and threat actor prosecution.

TL;DR:

• Do not underestimate your enemies.
• Know yourself better than your enemy.
• (Always) Include infrastructure in your security audit scope.
• Discontinuing or deprecating projects is as important as setting them up.
• Monitoring and detection is crucial.
• Decentralization must be real.
• Beware of TTPs and most used attack vectors.
• Pausing mechanics is still one of the most effective actions so far.
• Splitting funds among different smart contracts is also recommended.
• Stronger together.

🤝 The importance of security collaboration

In Babylon Finance, we believe that one of the best ways to improve DeFi security is to share relevant security intelligence (i.e. analysis, best practices and actionable countermeasures) among all the teams, protocols and users, as we all might benefit from a more secure, trustworthy and resilient ecosystem, resulting in a stronger DeFi security community.

For that reason, we started our security intelligence series by sharing our own security intelligence regarding an MEV Doom Loop which produced a series of liquidations in collateralized BABL positions. We hope that other protocols or teams can benefit from the analysis.

Today, we’re sharing lessons learned and subsequent recommendations based on the recent Axie Infinity’s Ronin bridge $600M hack.

💼 Executive summary

$600M (173,600 Ethereum and 25.5M USDC) were drained from the Ronin network bridge on March 23rd, 2022.

The threat actor managed to steal 5 private keys from 4+1 active validators where the later (+1) belongs to a discontinued third party agreement of Axie DAO — Sky Mavis.

Privileged actions that were executed by the threat actor required at least 5 out of 9 validator nodes’ signatures. The attacker forged fake withdrawals by using those 5 legitimate signatures.

It is still unclear but multi-factor authentication mechanisms could have been of great help to prevent this incident.

Unfortunately, there were not reliable detection mechanisms in place so the team discovered the attack 6 days later, on March 29th after one of its users reported an issue to withdraw funds. Private keys’ theft probably happened several days before the attack.

However, since the attack was not discovered for 6 days, the team activated specific countermeasures (i.e. pausing and halting) which are usually very effective for ongoing attacks to prevent new funds draining. They also involved top firms in the investigation and mitigation and reported it to law enforcement for prosecution.

Ronin is providing updates about the incident publicly however some information cannot be disclosed yet due to potential impact on the prosecution process. After recent attribution to Lazarus Group, they are planning to release a detailed post-mortem.

A few days ago, the threat actor began moving funds through Tornado Cash in chunks of 100ETH.

⚔️ TTP (Techniques, Tactics and Procedures) of the attack:

In the following picture, we represent the Ronin TTP in the MITRE ATT&CK matrix to better visualize the different tactics, techniques and procedures followed by the threat actor.

In yellow we represent those which we believe are potentially part of the TTP but not confirmed yet.

Ronin attack represented in the MITRE ATT&CK Framework matrix

Though still not certain, it seems that there were 2 different attack vectors:

Private keys from 4 legit ronin validators

They were probably theft by spear phising as it is the most common attack pattern however it is still unclear. Anyway it is unlikely that the 4 keys were part of the same vault or controlled by the same victim or server, but it is not impossible, for example, if it was not properly configured, audited or if was not using security best practices (as explained later).

Private key from an external whitelisted validator

It was probably obtained from the node itself. A potential back-door is mentioned, but we have to be careful with such word or statement. The are different types of back-doors but usually they are hidden and created by design. They can also be created by mistake from the dev team in the code. In this case it seems more related to an unsafe configuration rather than a development bug or vulnerability exploited. If it is really a backdoor, a patch should be released asap as it can be affecting several nodes.

On the other hand, the need to grab a 5th private key from an external whitelisted validator on top of other 4 validators indicates that at least 5 validator’s keys were not part of the same vault, victim or user, however it can also be part of a false flag to cover their tracks. It is also not-trivial to guess how the attacker knew the external validator’s key was still functional and part of the consensus mechanism, we should always know our infrastructure better than anybody else.

These details, together with “who sent the initial funds from a Binance wallet”, “how the keys were accessed” (which is part of the forensics investigation) and “why they had no additional authentication factors or safer configuration access” I am sure, they have been the keys for the attribution.

Several minutes before the attack started, the exploiter wallet received 1.0569 ETH from a Binance exchange’s wallet to fund the attack gas cost. I am sure that Binance collaboration has been of extreme importance to know more about the attacker or the funding manager however sophisticated threat actors usually do not expose themselves, they use “mules” instead.

Considering the stolen amount and the attribution to Lazarus, it has been probably a very sophisticated attack using advance malware or even a 0-day exploit through a spear phising (some of their TTPs) but we will see it in the post-mortem.

It is worth mentioning that after the attack relevant investors led by Binance are backing Sky Mavis, the Creator of Axie Infinity by a new $150MM funding round, to Restore Funds on the Ronin Bridge.

The attack only used 2 sequential tx’s calling the same withdrawERC20For function on March 26th to steal all 173,600 ETH and 25.5M USDC respectively.

Just few minutes later, the attacker used different accounts to swap all different assets into ETH by using 1inch and Uniswap and sent them back to the original exploiter address 0x098B716B8Aaf21512996dC57EB0615e2383E2f96 which had aprox. 175,913 ETH after the attack.

The attacker hold some days the funds in that account. Unfortunately some days ago it started to move stolen funds in chunks between 1K up to 3K ETH through other accounts splitting those bigger chunks into smaller chunks of 100 ETH before being transferred to Tornado Cash.

Attacker moving funds between accounts and Tornado Cash

🕵️‍♀️ Detection 👮‍♀ ️Response and 💻 Mitigation:

1. A user reported on March the 29th an issue to withdraw funds so the team discovered the issue right after.
2. Unfortunately, the mitigation started several days after the event occurred.
3. It is worth mentioning that the average dwell time (average time between the initial breach and the detection) is still 24 days (ref. Mandiant). In this case it is still unclear when those private keys’ theft took place, but probably it was several days earlier.
4. The team paused and halted different related services to avoid the same issue happening it again.
5. They increased the number of validators to 8 (out of 9) as a workaround. Few days later, all former validators were replaced and the team is planning to increase its network.
6. The team is actively monitoring stolen funds using top level third parties and exchanges.
7. As part of the response, they worked with top level forensic firms and law enforcement.

Evolution of average dwell time by Mandiant

🧠 Recommendations and lessons learned

1. Multi-sig is a great feature, but the problem is really how those signing parties are actually using their private keys.
2. The use and access to privilege accounts’ private keys should require MFA (multi factor authentication) instead of just stored in plain text or behind a simple passphrase (something that only I — as the signer- know). 2FA apps (what I temporarily know), use of HW wallets (what I have + what I know about the HW wallet) or even biometrics (who am I) are much better options and will probably become mainstream soon. We recommend to always use HW wallets even in multi-sig environments.
3. Using multiple signatures is a great choice but it is not enough as seen in this incident. If withdrawERC20For is a privilege function, it should be limited to specific multi-sig sender whitelisted address (i.e. smartcontracts like gnosis safe, governor + delayed timelockcontroller etc). It could potentially introduce specific delays on its execution (like governor alpha, bravo or OZ governor by the introduction of timelockcontroller).
4. Infrastructure security (i.e. versions, configuration, audits and maintenance for the whole infrastructure, even apps or private key vaults) is as important as smart contract’s security, we should never forget to see the whole picture.
5. Discontinuing or deprecating projects is as important as setting them up and their actions should be carefully executed and planned (i.e. what, when and how should we remove access).
6. Pausing mechanics is still one of the most effective actions so far for ongoing attacks. If possible, it should be ready and battle tested at least for specific functions (i.e.withdraw) to save as much funds as possible if the reaction is really fast.
7. Keep things simple and standard as much as you can. Some pausing mechanics’ flaws seen during last hacks like BadgerDAO’s suggest that it is worth to create long term de facto standard architectures.
8. Prevention, detection and monitoring capabilities are probably the most important but also the most underestimated priorities. Better detection does not necessarily means saving more funds (i.e. if the tx already happened) but it might increase the probability quite a lot to save more funds, to recover them or even save them all. We recommend at least to set some detection mechanisms by using simple alerts that can be set-up very easily and fast by using Defender from Openzeppelin and Tenderly combined with Telegram, Slack, etc.
9. As in any type of emergency, it is crucial to activate emergency services and partnerships as fast as possible to count with renowned expects to mitigate the impact. Trusted or preferred partners should demonstrate deep knowledge to be really effective if under attack.
10. Best effort approaches (i.e. calls and meetings) between partners should evolve into contractual SLA (service level agreements) where possible to guarantee enough resources.
11. The execution of periodic cyber-exercises to test detection and response capabilities is very important to avoid any delay in case of incident as well as to detect flaws or improvements in the process itself.
12. 100% security does not exist, but the more risks are identified and addressed on time, the more difficult or more pain for the attackers (in the highest levels of the pyramid of pain) so they might consider another different victim or even to give up. Risks management practices with an improvement plan is also one of the most useful best practices.
13. Splitting funds among different smart contracts is also recommended.
14. Partnerships are a key for the future of Ethereum and for any other network. These type of technology allows anyone to create effective detection and monitoring solutions because we all share the same network, so anyone can monitor and/or run forensics over immutable tx’s. We believe that in order to have a stronger DeFi security community we need to work and build together some new building blocks like enhanced detection and monitoring #strongertogether.
15. Fast, frequent, and transparent communication to affected users but also to the community (to avoid the same issue happening in other protocol) is always the best choice as Ronin’s did. On the other hand, users should understand that some details might not be publicly available during a law enforcement prosecution.

🛡Babylon Finance’s Best practices related to this attack:

However 100% security does not exist, in Babylon Finance (https://babylon.finance) we have implemented most of these recommendations which are described high level in order to give a better overview to our users:

1. In Babylon the Treasury is decentralized (owned by BABL — governance token- holders) under Babylon Governor (OZ Governor) and Timelock controller which is giving a delay of 7 days to execute any governance action. In addition to this, the team uses HW wallets combined with the team multi-sig in a gnosis safe.
2. All other related accounts (i.e. cloud infrastructure, github, dapp, etc.) are using MFA and encrypted vault and has passed a security infrastructure audit.
3. Privilege functions at Babylon are only using Babylon Governor time lock controller (7d delay) or team multisig (+hw wallets) as only valid whitelisted addresses.
4. All smartcontracts as well as all the infrastructure security is having frequent security audits.
5. We use to have a shared checklist for upgrades or new deployments but we also use it for closing initiatives and or governance proposals. At least 2 different team members must approve other team mate user action. Monitoring alerts are also set for the most relevant actions so anyone is aware of executions.
6. Pausing mechanics are following a standard for all smart contracts . We use a pause guardian which have the right to pause a specific smart contract or the whole protocol at once in only 1 tx (i.e. guardianGlobalPaused and guardianPaused can be seen in the BabController smart contract). Unpausing is reserved only for Babylon Governor or team’s multisig.
7. Prevention, detection and monitoring capabilities are not underestimated in Babylon. We have implemented different monitoring and detection capabilities based on alerts. We also implemented some automatic counter-measures for specific processes. We use Defender from Openzeppelin, Tenderly, TG bots, enhanced by our own custom scripts.
8. We are investing in having great partners as well as setting communication channels with all of them. In certain cases it has been of great help (i.e.as seen during MEV Doom loop). Anyway we feel that there is still room for improvement. As an example, we asked to get some SLA agreements with some security audit firms and they do not offer such services. We will continue improving our partnership network.
9. Being Babylon an active community with different launch events, it is bringing some kind of exposure to malicious actors. As of today, it helped us to test our capabilities and to improve our detection and response processes. As a result, we are better protected and will be better in the future, however 100% security does not exist.
10. We have a continuous security and risk management process which is helping us to improve along the way.
11. Babylon Finance architecture by design is splitting funds among different smart contracts (gardens, strategies, other protocols investments or vaults, rewards, etc.) so it is unlikely of having a big impact by only one tx as happened in Ronin’s.
12. We use to communicate fast, frequent, and transparent to our users using our Discord channel.

Conclusion

As a conclusion, we should never underestimate our enemies especially when APTs groups are behind. For that reason it is also mandatory to know our infrastructure state of play and to have a continuous audit that might actually help us to improve anything that can be improved.

As “Sun Tzu said in The Art of War”:

“If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle”

There should not be anyone knowing more about us than ourselves. Beware of insiders because of this.

We hope that some of the recommendations and or suggestions of this post can benefit other projects, and in particular, the future of Ronin’s infrastructure.

We wish to all users, team and all partners involved in the incident the best of luck. Fortunately, some stolen funds are still in the attacker’s account and top level partners are working with the team so there are still some opportunities to recover some funds.

If you are interested to build together a stronger security community, please reach out 👋, and let’s chat about it on our own Discord

🛡🛡🛡🛡

If you haven’t had a chance to use Babylon yet, please head to the website and take a look. We already launched publicly!.

You can join our discord community here.