🤖 The MEV Doom Loop
Babylon is a community-led asset management protocol that enables users to create investment clubs (we call them gardens) and invest in DeFi together. It’s built on the Ethereum network and it’s non-custodial, transparent, permission-less, and governed by the community. BABL is the governance token behind it.
TLDR: A perfect storm and MEV doom loop. MEV bots triggered a series of liquidations in collateralized BABL positions which caused more sales which in turn triggered more liquidations which would cause further sales….
In this post, we are going to cover the events of last Thursday that caused BABL to crash from $50 to $11 in less than an hour.
🛫 Boarding a Flight
On Thursday around 1 pm PST, three members of the Babylon team were saying goodbyes after a long and peaceful retreat in the pacific northwest. We boarded different planes in Seattle towards San Francisco, Chicago, and New York. It was a great time together and we were all relaxed and excited about the future.
Little did we know of what was ahead of us…
My plane took off and I was waiting to get access to the internet. I was planning to write a few more tests and maybe watch a movie afterwards.
🚨 Crisis minute by minute
Around 1:43 PM PST, the team starts noticing strange behavior. Here you can see extracts from our Telegram chat 🚨 (Times are in GMT-5/Chicago)
While I was offline, the rest of the team was seeing many 🤖 MEV transactions that involved our Fuse Pool and looked like the following:
https://etherscan.io/tx/0x02567d5e36f31293e4baac4dfe800ad99d420693cf7c3beb4a45963dfc81f73d
https://etherscan.io/tx/0x3421bb8e1e029cb286e5e159505c809d267871c7646f89ed9eaeaf5f83aea115
Quick Aside about MEV Bots
🤖 Monsters in Ethereum are real , they are called MEV robots. In our case, they were indiscriminately sending these transactions one after the other to earn liquidation fees. MEV Robots don’t care about your feelings, your users or your state of mind they just relentlessly extract every ounce of value that they find within the Ethereum Dark Forest.
At the same time this was happening, the team was also seeing rebalances of our Liquidity position in Uniswap v3.
At this moment, exactly at 2:06 pm PST I get back online. It was spotty and slow but it was semi-working. Then…
I frantically started to review transactions as fast as I could on Etherscan, which wasn’t really fast given my internet connection. If something happens, pray to God you are not stuck in a flying shoebox for the next 5 hours with crawling internet speed…
First of all, user funds were always safe. BABL price was the only thing affected. Under a bit of panic and a lot of adrenaline, our erroneous assumption at that moment is that maybe Gamma got hacked. Context: We are using Gamma to concentrate liquidity and improve the efficiency of BABL-ETH trades.
At this moment, I reach out to Gamma and we start looking into it together. The transactions that remove liquidity also added it, they were rebalancing and not removing liquidity. The rebalances were caused by the sale pressure triggered by all the fuse pools transactions above.
Example of a Rebalance
https://etherscan.io/tx/0x9daf4d10844cf109dd4f4ad4e9e9a2efa684e1c4e53af0fd96069f27d98aca08
At this moment, we start looking a bit more deeply into the fuse pool transactions. My internet makes every page load extremely slow to load and my cortisol is through the roof 😠
I manage to get my hardware wallet working in the middle of my economy seat and send four transactions to our multisig to disable all borrowing markets in our Fuse Pool at 2:26 pm PST
First tx: https://etherscan.io/tx/0xe494ce146eeff04f7298a230317b84e580111df8f218eba718e678e170f40a7f
After a few minutes, we realize that the issue isn’t solved, more MEV transactions are happening so we keep looking. We pull in the Rari team to help us look into it. In the meantime, I ask them how to pause the minting of new collateral because it is not available through the UI.
We do that on BABL through the following tx:
https://etherscan.io/tx/0x4fc9be5900fa6ae2b92c7bdda3ed3fd3fc27751b05636195acda2a9584c51675
A few minutes later, the MEV liquidations keep happening…
https://etherscan.io/tx/0x5b847b58d208e2388803409b1eb072cdc4877c3a00847a60d5c0e4fd0334545b
I asked the Rari team to see if we can pause the liquidations in the Fuse pool until we figure things out.
At 3:31 pm PST and around 2 and a half hours after the 🤖 MEV doom loop started, the liquidations were finally stopped.
https://etherscan.io/tx/0x53431f5ccb95c72025394f7da4fb9aca5ca1467845bf0beb411c182967d2c15a
Given all that was going on, the shitty internet, the flights, and the complexity of the interactions, the team responded fairly quickly.
However, a lot of damage was done:
In just 2 hours, more than 6.5k BABL was liquidated from the Heart of Babylon and its strategies. The Heart of Babylon has collateralized BABL positions to borrow stables and ETH to generate yield and distribute it back to BABL stakers. More on this later.
⚰️ Resuscitating BABL
Now that the bleeding stopped, we needed to figure out how we could resuscitate the price of BABL before restarting the fuse pool.
Every second counted because many people were now taking advantage of the sale and they were sniping BABL at 20% of its value just 3 hours ago.
The more people that buy at the low, the more people that would be selling it once we bring it back up and more funds that we would need to bring it back up again.
The team decided to step up and dedicate ~$200k to this purpose. However, we had to be smart to make the most out of our funds and prevent our buys from being arbitraged once again by cold, impassive, 🤖 MEV robots.
Here is what we did:
- We instructed Gamma to remove our concentrated liquidity to increase price impact. At 6:33pm PST, the concentrated liquidity was moved out of the way https://etherscan.io/tx/0xa7327977b343308d7ebb8f673bda39528d798662693696211fa5b2c37dd8be7d
- Send txs in small batches through Flashbots to prevent the transactions from being observed, frontrunned and arbitraged away. From 6:33pm PST until 8:21pm PST.
First, we sent 100 ETH from the team multisig to a quick account:
https://etherscan.io/tx/0x2974298a1ccbe3b94c993a892946b51085ac2c09fbc4c9aef5b9c3bb35f3e829
Then we started purchasing BABL in small amounts. The goal was to restore the price to the level before the cascade started ($48) plus a bit more anticipated some extra sell pressure.
We spent a total of 23 ETH in those two hours buying 1200 BABL. Price after the last transaction was $55.46. https://etherscan.io/tx/0xd501023c249a0464a789ad68b25e06f2b18d2371f6c71c3a0ff0232010b13145
3. Then, we restored back the liquidity. At 8:43PM PST, concentrated liquidity was back up.
4. We restored liquidations, borrowing and minting in the Fuse pool around 9:30pm PST.
5. The last step was patching and mending the 🫀 Heart of Babylon. The two strategies suffered more than 6.5K BABL in liquidations.
At 10:40 pm PST, we finalized both heart strategies. The next day we would top up the strategies and mining rewards to cover the liquidations and make all the BABL stakers whole.
⏰ 9 hours later everything was back to a stable place.
📖 How did this happen
Now, that we have recounted the events of that evening minute by minute, let’s dig in and see how it actually happened.
Aside from the perfect storm timing that caught the members on a plane and exacerbated the issue, the root cause boils down to three reasons:
- Thin liquidity on BABL-ETH.
- Strategy taking more leverage than we thought via Fuse pool.
- Lack of real-time alerts and monitoring on fuse positions.
BABL-ETH Liquidity
First of all, it is important to go back to the end of our NFT IDO event, The Prophets. After the event concluded, we created the Uniswap V3 partnership with Gamma and deposited approximately $2M in liquidity on Nov 30th 2021.
ETH was worth $4600 at the time in what turned out to be local top and the worst possible time to buy ETH and add liquidity. The subsequent bear market that dropped ETH by half and has consequently reduced the price of BABL and the TVL of the liquidity.
Starting January there was also considerable sell pressure from the mining rewards. Until The Heart Of Babylon was released on February 3rd, there was no upward price pressure or staking mechanic to counteract this imbalance of flows. These two factors combined hafe reduced the pool TVL to only $750k.
🧮 35% vs (35% of 55%)
The 🫀Heart of Babylon takes collateralized BABL positions on our Lending/Borrowing markets to borrow stable coins and generate yield from them, which in turn will be converted back to BABL. This is one of 11 mechanics that create value for BABL stakers.
On February 9th, two strategies were created and executed in the heart:
- Lend BABL, borrow ETH to stake it in Lido.
- Lend BABL, borrow DAI to enter CRV Frax Yearn Vault.
https://etherscan.io/tx/0x0a2b26ef2c4fea25f99628acb1f154c1a2a32e0875f4a2c9a055d057e3d0da21
https://etherscan.io/tx/0xa91b342cd6c16ca603b1ac5421f9ded36f8f8e65b80ea992e85e6e183a6fd0c9
When the two strategies entered, BABL was around $72 and they both minted about 10k BABL of collateral.
Now, it’s really important to understand that BABL collateral factor in our fuse pool is only set to 55% to ironically limit the potential for liquidations.
Look at the following snippet of code that calculates the amount to borrow in a Babylon Strategy.
Now, borrow strategies in Babylon borrow only 35% (maxCollateralFactor) of the capital added as collateral to the strategy. However, there is a big caveat, most of the markets in Compound and AAVE have collateral factors that ranged from 70% to 90%. This means that a borrowing strategy in Babylon even if the price of the collateral is halved.
However, BABL collateral is only 55% which means 35% if you borrow 35% you are actually borrowing 64% (35% of 55%) of your max amount.
The code above worked from the assumption that assets have a collateral factor of 100% which is more or less true for main Compound and Aave markets. However, this meant our strategies were flying really close to the sun. If BABL dipped below $44 they would be partially liquidated, which they did.
Everyone of these liquidations, sold around ~200BABL which given the thin liquidity explained above, put the positions again ready to be liquidated. Rinse and repeat, feast for the 🤖MEV bots.
The better version of the code above should use both parameters. We have already submitted this patch and new versions of the heart will use it.
borrow amount = capital * max borrowing factor*collateral factor of asset
🕐 Lack of Fuse Pool Monitoring
We have a lot of alerts set up in Tenderly and OpenZeppelin defender to manage our smart contracts and events they emit. However, we hadn’t set up any for the fuse pool, assuming the Grafana dashboard was enough for us.
Had we known that the strategies were close to liquidation, we could have taken several steps to mitigate the issue like finishing the strategies that moment or purchasing BABL to increase the price and prevent liquidation.
Moving forward, we are going to set up both an automated market maker that will protect the price from getting close to the positions of existing heart strategies along with detailed alerts related to the positions that use BABL as collateral.
💪 Conclusions
You never let a good crisis go to waste. We have learned a lot from this episode and we are going to take the following measures to prevent it from happening again:
- We need a lot more liquidity on BABL-ETH. Our pair cannot be so thinly traded than a liquidation would cause the same position to get liquidated again. BIP-14 has been sent to governance to add extra $200k in liquidity from the treasury.
- Prioritize the release of the bonding program as much as we can, that way other Babylonians can help us add depth to our liquidity pair and get rewarded with a discount on heart tokens (hBABL).
- At the same time, we need to be able to actively manage liquidity quickly during a crisis. To this end, the treasury is also going to send the Gamma LP tokens to the team multisig through BIP-14 if approved.
- We fixed the bug in our borrowing integration and now the heart will at most borrow 40% applied to the collateral factor of the market. Instead of 35% blindly applied to the capital irrespective of the collateral factor.
- Set automatic buybacks when the price dips below a target. Monitor the liquidation price of heart strategies and have an auto task in Defender running to purchase BABL in case the price gets close to liquidation to buy time for the team. Trigger alerts in our TG channel when this happens.
That’s all for today. Babylon was not hacked, funds are safe and we are on track to a big public launch later this month.
🙏 Big appreciation to Gamma and Rari for helping us in the war room
🙏 Babylonians for keep supporting the project day and night. We wouldn’t be here without you.
If you haven’t had a chance to use Babylon yet, please head to the website and explore the different gardens available. You can visit Babylon here.
Join our Discord and ping us if you have any questions.
