Anchor Protocol: How the dev team rigged liquidations

Background

7 min readAug 15, 2021

We noticed suspicious on-chain activity from the Anchor Protocol dev team’s personal liquidator about four weeks ago after block height 3757709. After monitoring on-chain transactions from the liquidator for a week, it became clear that the dev team had granted their liquidator privileged access to the price feed, and the liquidator was aware of the price feed bot’s state off-chain. We also noticed many faked liquidation attempts by the dev liquidator to obfuscate on-chain activity.

On the Anchor Protocol discord #dev-chat, I asked if the dev liquidator had been granted privileged access to the price feed on 20th July. The usually responsive dev team never responded, but the dev liquidator altered its behavior since then and started spamming about 20,000 bad transactions on-chain in a further effort to obfuscate the actual on-chain activity.

Privileged Access

Since we use the term privileged access, I want to specify what we mean by it. Privileged access, in this case, is an advantage in getting the price solely because the same team runs both the price feed bot and the dev liquidator. All of the scenarios below should count as privileged access:

The price feed bot and dev liquidator is now in the same process, and the two wallets are sending the price_feed tx and liquidation tx sequentially
The price feed bot uses inter-process communication to tell the dev liquidator the price immediately after sending the async tx
The price feed bot and dev liquidator are in the same server with a local node, the price feed bot send the tx to the local LCD, and the price feed bot reads the local mempool, which is guaranteed to have the price_feed tx first
Some contrived attempt at decentralization theater which still guarantees the dev liquidator is the first to read the price from the mempool, IE:

On-Chain Analysis

In case you aren’t familiar with the Terra and Anchor Protocol liquidations, let’s establish some facts to get us on the same page:

The current Terra version is Columbus-4
It uses Tendermint 33.9, and there is no transaction ordering. All transactions are processed in a FIFO queue by the mempool
Frontrunning a transaction that is already in the mempool isn’t possible since the mempool is processed first-in-first-out
Backrunning the price_feed tx rarely happens because transactions can’t be prioritized
The Anchor Protocol oracle is updated by a single trusted address controlled by the dev team
The dev liquidator is terra18kgwjqrm7mcnlzcy7l8h7awnn7fs2pvdl2tpm9
The price feed bot is terra1zue382qey9l5uhhwcwumjhmsne49a0agwhd60d
Detected suspicious activity around block height 3757709, and contacted dev team on discord around block height 3816781
We assume that validators are not colluding to reorder transactions in the mempool to extract MEV for the dev liquidator
The data presented here is reproducible with open-sourced code available in this github repo. Fell free to look through the code and independently verify the data

After block 3757709, we noticed that the dev liquidator started backrunning the price_feed tx in most liquidation attempts, while historically, that happened approximately 10% of the time. This increase in backrunning suggested that the dev liquidator received the price off-chain, and the price_feed tx and liquidation tx were sent consecutively. To test our intuition, we analyzed all liquidation attempts by the dev liquidator and plotted them with matplotlib.

In our code, we assume any liquidation transactions processed immediately after the price_feed tx to be backrun and all other liquidation transactions to be normal.

Immediately we notice the noise generated by the dev liquidator after we contacted the dev team. Instead of letting those 20,000 bad transactions cloud our analysis, we will ignore them because it doesn’t matter what the dev liquidator did after we contacted the dev team. If we can prove that the dev liquidator had privileged access to the price between blocks 3757709–3816781, then we know that they aren’t an honest actor. So let’s run liquidator_stats.py, which now stops at block height 3816781.

Now we plot the same graph of other liquidators taken from Extraterrestrial Finder’s liquidator list and compare the dev liquidator to other Anchor Protocol liquidators.

Finally, let’s generate some backrunning stats for all liquidators.

Before block 3757709, the most performant liquidators usually backrun the price_feed tx from 10–12% of the time.

After block 3757709, the dev liquidator is backrunning the price_feed tx 83.82% of the time, implying an order of magnitude speed increase. Our internal testing, using a sample size of a thousand transactions over a week, shows us that the latency required between two transactions to backrun each other 84–90% of the time is likely at the inter-process communication level or faster.

For us internally, the increase in backrunning from 9.18% to 83.82% is concrete proof that the dev liquidator now has privileged access to the oracle price. But for this writeup, we are not claiming that the dev liquidator is a dishonest actor based on those statistics because of the complexity required to verify that claim and since a much more rigorous and easily verifiable proof exists.

Proof of Corruption

Let’s assume that the dev liquidator is an honest, unprivileged actor and that validators aren’t colluding to extract MEV for the dev liquidator by reordering transactions.

The fastest way to liquidate is to read the price_feed tx from the mempool and send the liquidation tx. Since the Tendermint 33.9 mempool used in the Terra Columbus-4 chain is a FIFO queue, the price_feed tx is already queued ahead of any liquidation attempt and can’t be frontrun.

Now, let’s run find_frontrun.py to check if the dev liquidator frontruns the price_feed tx between blocks 3757709–3816781. We get the following output of frontrun transaction hashes:

02A9CB9B9F37ED0025608D130BDBF407415E7E7B4A7D5AA89BA1860D3B96D49B
BB4CD51AF327A64764D7B658B2CB7147388A376F138FA046E2C4BECFF9502385
CD766ABD189DCBD0B60EA3E2B3B865B6219BE313D1FAA33C22B5C14F609BACA7
D1B1F2C7CCF7F4D4D8734C367C29FDDE8A97F4EAEEBE99AB34A127104F2A18DC
4784810832D709CACEBC7798D35A6D06520C8207D0FE0D2A45AE5DBD95227EB5
DC04AA563A87D15ED7B711887D9327181BFE4225CC242B1F3EA109581E318EFD
D51103D7A914B13F94098B22B9F8B6A2C675C918CE5ABB818286116874483B67
B4E6A255B8EC84F3738FDB8F7A7350611BF91FAFD7D6BA4C1FCB1D2AD1B53FB7
90890842BDD27979281DD6FE45A4ACDA5FD2C7B314A604A7130945B77FAA760C
22D68AE5EF180D5196C3FB3B241645E32006324A7E179F02F0799A4185116070
8995FA7A5F31EDA40DE71231A56585332E287CEBC3921F3A6016EC6048A36D76
8086A6170370B0C328D009869B227691B424BFD9BF8DDBE1ACBB5E871AA75704
8BAE8A21C452FC2D43737B9AF0C20193BB56D0522E5081A3E7AC5494A1179D63
25B499B677E4A2A5F600F0D84BAC8D0CD2CF7D32220AD54FB3836BAEBB257A94
5F0B930992D709F4DF5AA625788900BB111763885FFB809F6413EFF0AF80D60E
93A85C9764CBF7769A8098930D6C66462FF0415DF52B6CA9C252794F0D9BC5A0
C35EDA470A692B82EF95C1D61D57E312B2D818B561E01D66FC8B0BC69F27DCB1
9854C7876077A99ED5615A121E8382DA904F7F6A5FA56D53CF6FE13DA2B44C68
0DAF476AE183A1326843AA4F785E5A0CC6732FF5881CD2BAB29549A28E0EC4E9
AD32323E2C39EE73EAA0E9063E6CF35315170D7646FF6E24B44F1C1DD0BA6F51
874715BB8104893159F44CA4E92912EB94F988519BCF67D4825E05690D675C6E
7073747D46734AD9B4CC6CFDC81EE87B67EDCA5D57AE255EDC525EB2995546C8
784C3FCE1C89547E0D4A9909F221454EDFF03796A1894718578E165A3F3ECDE0
4BBF3CBE7FAC5FADEDC283AF5F70872D55DF6B13A3A9C38D41D687899D8DD334
A65DDA6F9420EF2FC01D771A23D26E6D1973E74ABB8293798AAB2F962824762E
392E785ED11773C9B23F6284FA197700DC8FFB215998C940C1D83E4D01AC0322
FEAFFB04727D5DCA1AA2944560F2F5B28743CE2A17755B70E5E2FB9EF72F0BB5
550A6F8D367821A7C3992F1EBA4960E6803480698C53343097ADAE766E89931B
7445E51B8AC8DA194ABBFEAD5A6B7A559A59E10CEAC8B45EC910F20FFA2C0D67
DDC55B137EF743AB366DA660EE2DE91347331F004983417059632EE30C621202
593B5678DF873D0BB704DF043D4189E894162CAB13A4C979CB7D4946D89452D9
1A9E7D59B8AC528D43E2C2F2793075F762C3ABFF05ED66531E71B1018C3B1737
99F71D472BDB65BB1DA858B9D50F2F259A3647A110C91B4E5D00939A55339099
F2BA1B52F093C22E6303DA784BD079E3CE17999F8F4871E3930C05B30D46CA71
0521024E3841E4DC53A1261B19A2050D05ADC613FD8428EBBB7D7DA7B4535070
938C8A529B12EE5E46B273276DDAD41B93CFEB5C27AFE8938B249025F68A2E0B
B94058EDD57A9C16EBF176D4C06D77181AA7CB8FDE620C94B62C0E72EAD5F4FD
AA65894087DACEB78A9E0EE62F532CDC1CA35177FD919056E8894954A3E8D893
D406E72D763E4EECA272A93C99283550FFA0BC148A79BD798EEC36AD21AC8FA7

Now we can put even more granular detail on the dev liquidator stats between blocks 3757709–3816781:

backrun: 855
normal: 126
frontrun: 39

How is it possible for the dev liquidator to frontrun the price_feed tx if it’s reading that transaction from the mempool? There are only two scenarios:

Scenario 1: The dev liquidator is randomly spamming liquidation transactions without waiting to read the price_feed tx from the mempool, and sometimes those transactions end up right before the price_feed tx
Scenario 2: The dev liquidator is not reading the mempool. Instead, the dev liquidator is getting the price off-chain directly from the price feed bot, and both transactions are sent async consecutively. Since Tendermint nodes use a goroutine per transaction, it can’t guarantee that those two async transactions are processed in order all of the time, and sometimes accidental frontruns happen

Just looking at the dev liquidator stats, we know that scenario 1 is probabilistically impossible. Still, we conclusively rule it out by manually checking the liquidator’s behavior before and after the frontrun in the link below. We include links to both the transaction and the block to make independent verification as easy as possible.

https://throwaway0xfr.github.io/frontrun_analysis.html

The frontrun_analysis page shows that the dev liquidator was perfectly backrunning the price_feed tx before and after the accidental frontrun, so we know that the liquidator was not randomly spamming transactions but is instead precisely reacting to price changes.

Since the dev liquidator can’t read the mempool for the price_feed tx and frontrun the same price_feed tx, it is getting that price off-chain, and the only credible explanation is scenario 2.

Conclusion

Based on this evidence, it is clear that the dev liquidator is a dishonest actor, and the dev team is abusing its privileged position of running the price feed bot.

Please direct any questions, comments or criticism to https://twitter.com/0xfr_