May 19th Market Volatility
Synopsis of Events and The Plan Going Forward
Update: Kava Labs launched a rebranded product portfolio in late August 2021, and the following archived article contains outdated information. Application rebrands on the Kava Platform are as follows: Kava CDP is now Kava Mint; Hard is now Kava Lend; Swap is now Kava Swap. To learn more about the rebrand, please read the announcement.
During the recent period of volatility in crypto markets, a number of users of the Kava webapp (CDP and Hard) reported atypical delays when trying to transact with the Kava blockchain. For some users, this resulted in them being unable to repay or re-collateralize borrow positions, which caused them to be liquidated.
This article will provide a summary report on the issues we encountered along with the actions Kava Labs engineers are taking to prevent this situation from occurring in the future. In addition to this report, we are working on a SAFU Fund proposal to compensate affected users.
Synopsis of Events
The crypto market experienced a period of rapid downward price movement starting around 01:00 UTC on May 19th. During that time period, the Kava blockchain experienced increased usage as users, arbitrageurs, keepers, and oracles began transacting more frequently in response to the changing market conditions.
As price movement accelerated downwards around 14:00 UTC, a keeper bot began spamming the network with zero fee transactions that attempted to liquidate users’ positions. While most of the transactions failed due to the positions not being liquidatable, the mempools of nodes that accept zero fee transactions rapidly began to fill.
At the time the Kava webapp was connected (by default) to nodes that accepted zero fee transactions — giving users the option of not paying fees at times of low congestion. Because of this configuration, the mempool used by Kava webapp users became backed up, causing delays in transactions up to 30 minutes. As a result of the delays, many transactions that attempted to re-collateralize borrow positions in Kava and Hard protocols did not succeed in time to prevent liquidation.
Upon being alerted to the congestion, Kava Labs engineers updated the frontend to place transactions with fees in a separate mempool, relieving the congestion and allowing transaction processing to proceed normally. The total time of exposure between the event beginning and our rectification was two hours. During this time, users with access to alternate full nodes were able to successfully broadcast transactions — i.e. the issue was confined to the frontend and not due to a protocol failure.
The Plan Going Forward
In addition to the steps taken above to alleviate the immediate issue, the following longer term steps are being taken to ensure greater protocol stability during periods of volatility:
- Replacing the current FIFO (first in = first out) mempool logic with a priority queue where transactions are ordered first by fee, and then by gas wanted
- Improve frontend gas estimation to reduce the average gas consumption of transactions
- Strengthen internal alerting systems to expedite notification to Kava Labs engineers when the mempool is experiencing elevated congestion levels
- Take advantage of transaction deadlines in v0.43 of the Cosmos-SDK to make it easier for users to replace stuck transactions
To make affected users whole, Kava Labs has conducted an analysis of the on-chain liquidations that occurred and the liquidation penalties that were assessed. Because it’s not possible to reconstruct the historical mempool state, this analysis includes all users who were assessed liquidation penalties in the period from 1:00 UTC to 16:30 UTC on May 19th. Using this analysis, we will be issuing a SAFU fund proposal that refunds users’ liquidation penalties.
Implementing the Plan
First, we need to introduce a new community spend proposal message that can spend to multiple addresses. Next, we need to create a proposal that specifies the payout amounts for each user. We estimate that this process will take two weeks. Ultimately, the community will have final say in voting on the SAFU Fund proposal.
As always, the safety and security of user funds is our top priority. We have made many improvements to the backend to handle these inevitable periods of volatility, and overall, we were pleased that our infrastructure and the blockchain itself remained stable throughout the day. Further, the creation of an on-chain SAFU Fund has proved to be a timely resource, and should give Kava users peace of mind that we are committed to the long-term growth of the platform and ensuring that users’ concerns are dealt with in a fair and transparent manner.
- Follow us on Twitter: https://twitter.com/kava_labs
- Join the conversation on Telegram: https://t.me/kavalabs
- Checkout our code: https://github.com/kava-labs
- Subscribe to our YouTube: https://www.youtube.com/c/Kavalabs/videos
Disclaimer: This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making investment decisions.
