Gauntlet’s Parameter Recommendation Methodology
There has been a lot of great discussion in the forums around our parameter recommendations, and we wanted to provide updated context on our approach to recommending parameters for protocols we service. Do note that we are constantly evolving both our methodology and our simulations to make them more accurate to better support protocols we service.
The goal for our Risk Management product is to balance Capital Efficiency and Risk to the protocol. All else being equal, our models err on the side of increasing Capital Efficiency as long as the amount of risk in the system doesn’t increase significantly (relative to the protocols reserves).
This is the first part of a pair of blog posts diving into more detail around how Gauntlet’s Platform works. This post covers how we make parameter recommendations, and the next post covers our model methodology in detail.
High-level methodology
Gauntlet automates risk parameter selection by framing it as an optimization problem, where the objective is to minimize expected insolvent debt and liquidations (a proxy for borrower funds lost to liquidators) and maximize Borrow Usage (a proxy for protocol revenue). We use agent-based simulation models tuned to actual market data to predict the impact of alternative parameterizations in order to evaluate the objective function at other points in the search space.
Agent-based simulation (ABS) is a powerful tool for analyzing the behavior of complex systems as it allows one to break down the problem into modular components that are easier to reason about and can be independently tuned. ABS has been used in a variety of stress test contexts, including estimating censorship in cryptocurrency protocols, detecting fraudulent trading activity in CFTC exchanges, and stress testing frameworks from the European Central Bank, and the Federal Reserve. These simulations, while powerful, can be difficult to make both useful and accurate as model complexity can make it hard to match experimental results. Careful design, tuning, and infrastructure architecture can help avoid these pitfalls and has made ABS invaluable in industries such as algorithmic trading and self-driving car deployment.
In such industries, one takes care to ensure that the simulation environment replicates the live environment as closely as possible. Agent models interact with the deployed code in a live environment to enforce replication. This minimizes errors from mistranslations or missing minutiae. While the infrastructure overhead of simulating users interacting with a piece of complex software can be heavy, it ensures that errors are limited to those in models of agents as opposed to errors in the models of system dynamics or from transpiling business logic.
The Gauntlet platform includes a Python domain-specific language (DSL), akin to Facebook’s PyTorch, for defining models of user behavior (e.g. borrowers, liquidations, etc.) as well as an interface for running simulations directly against Ethereum smart contracts using a version of the Ethereum virtual machine. In this environment, agents can also interact with additional on/off-chain systems such as an exchange.
We use this DSL to define a set of risk scenarios to stress-test the protocol’s “portfolio”. This involves running thousands of simulation instances daily over a cluster of machines and aggregating the results to compute the key protocol risk metrics. Gauntlet externalizes these results via our public dashboards. We then repeat this process for other selected points in the parameter search space to optimize the objective function (explained in more detail below) in order to select a new set of recommended risk parameters for the protocol.
Deep dive
As mentioned in our previous post on VaR, the biggest risk to a protocol is insolvencies. On the flip side of Risk is Capital Efficiency, and we strive to achieve a balance between the two.
Let’s go through the life of a parameter recommendation:
- Our ETLs (Extract, Transform, Load — our data pipelines) pull in current market data and borrower data as well as core liquidity data. You can see some of the key pieces of data on our dashboards under Market Statistics.
- Our simulations then run with the current risk parameters in the system (i.e. LTV + Liq threshold/CF, Liquidation bonus, Reserve Factor etc.) in order to understand the current risk and capital efficiency in the system.
- After this our platform runs many more simulations, iterating over different possible parameterizations. Their goal is to maximize an objective function that captures the core components of risk and capital efficiency. This objective function is robust in that it encapsulates the core concepts we care about (insolvencies, liquidations, and borrow usage).
- After our simulations have found a set of parameters that optimize this objective function, our Data Scientists then go through a round of manual QA to ensure the quality of the recommendations and make sure that nothing crazy is being recommended.
What metrics do we care about
Our simulations model what happens in ‘catastrophic’ scenarios by strongly incorporating the current state of the market and users of the protocol as inputs, and then relying on rational behavioral and elasticity assumptions to define agents that allow us to ‘observe’ what would happen.
Once our simulations have run, our optimization model’s goal is to understand the ‘end state’ using the metrics we have previously described (as well as some other intermediate metrics). The core concepts representing risk and capital efficiency are:
- Insolvencies
- Liquidations (a proxy for capital lost to liquidators)
- Borrow Usage (a proxy for protocol revenue)
The below table describes these in more detail and also describes our goals in interpreting them.
It is important to note there is no stable setting for risk parameters for a given protocol since user behavior, liquidity profile, and market volatility are always changing (hence why we manage these parameters dynamically). At its core, the above metrics are probabilistic in nature, especially when considering the specific effects of a parameter recommendation (simulations are forward looking ‘what if’ scenarios).
Why simulations
The need for simulations for this rather than heuristic methods cannot be understated and is the reason that this technique is heavily used in other industries. We commonly get asked — what specific market/borrower data is driving your recommendation to i.e. increase collateral factors for ETH this week. This is a tricky question to answer as most of the time the answer is ‘a bunch of factors’. We can go through an example of a particularly tricky interpretation:
The above heatmap is the WBTC Net Insolvent Value % output in Compound from our simulations on 2/11/22. It seems unintuitive — why would potential insolvencies increase when collateral factors decrease (high collateral factors should generally be more risky)?
It is hard to walk back directly from specific market or borrower data to this outcome. In this case, real borrowers have supplied multiple assets as collateral and if the WBTC CF were to be decreased, these large accounts became more sensitive to other asset price movements which led to more insolvencies (in simulation). Importantly though, this does not mean that CF should be increased or decreased directly from this result/heatmap alone.
This is an optimization problem that needs to weigh the relative borrow increase against the possible additional cost of insolvencies. Gauntlet’s objective function formulation is something we have spent a lot of time building and optimizing to manage this tradeoff. Our dashboard speaks to the core components of the objective function by showing the relative increase in Borrow Usage against the increase in Value at Risk and Liquidations at Risk, as well as the entire distribution of these statistics. We will continue to iterate on this so the community is better informed.
At its core, this question speaks to a common issue in AI/ML systems (or complex systems in general) — explainability of models. Complex systems, like Gauntlet’s Platform, will find it hard to find specific drivers that justify a given output/parameter recommendation due to the fact that there are many ingested features that drive our simulations in non-linear ways.
The real success criteria occurs during actual catastrophic events like Black Thursday in 2020 or even just these last few weeks. In these times we can empirically look at the resiliency of the protocol, tune our models with more data, and determine how well our simulations have done at predicting the security of the protocol (in the ML world this would be measuring on real unforeseen data instead of training/validation sets).
We have a great case study these past few weeks. While we won’t go into all the specific parameter changes we have made, we have largely been raising the capital efficiency of all of the protocols we work with over the past few months. We are happy that in light of this, none of our clients experienced any major insolvencies even though some assets crashed by more than 50% (see our posts on Compound and Aave for more details). This is the value of Gauntlet’s Platform.
We hope that this post helps understand how we make parameter recommendations at Gauntlet. We are constantly improving our methodology and will continue to update the community as we make changes that improve our recommendations.
Check out our follow up blog post for more detail on our model methodology!
If you found this blog post interesting and want to work with the awesome folks at Gauntlet, reach out and apply at gauntlet.network/careers! We are hiring across the board
Special thanks to Watson Fu, Hsien-Tang Kao, Rei Chiang, and Paul Lei for their contributions
