Circuit Breakers: A Chainlink deep-dive
In a recent engagement, Arbitrary Execution’s audit team worked with a protocol that interfaced with several external price oracles. Citing an article that made rounds when the TerraUSD Stablecoin (UST) lost its peg, the protocol asked us for strategies to guard against extreme price behavior in external price feeds. During our research, we came across Chainlink’s internal price feed circuit breakers, which can pause a feed without warning. The documentation on circuit breakers is sparse, and this article aims to fill in the gaps.
Background
The cited article describes how the protocols Venus Finance (BSC), and Blizz Finance (Avalanche) were affected when Chainlink price feeds paused unexpectedly as the price of Terra (LUNA) crashed.
Terra (LUNA) and TerraUSD (UST) are tokens created by Terraform Labs. TerraUSD is an algorithmic stablecoin, whose companion token Terra could be burned in order to mint TerraUSD when UST moves off its 1:1 peg from the US Dollar. To summarize the full timeline of events, UST lost its peg to the dollar and fell to approximately 35 cents, and LUNA fell to fractions of a penny. These extreme price events caused the Chainlink LUNA/USD price feed to pause.
Blizz Finance was drained completely for approximately $8.3MM, and Venus lost $11.2MM during the price feed outage, leaving many users in the community with questions for Chainlink. After several users posted questions in Chainlink’s Discord server, the team posted an official response. In the statement, they explain that price aggregator contracts have “circuit breakers” to protect against flash crashes and add that protocols were notified when the feeds were paused:
This response provided some useful context, but still left us asking questions about these so-called “circuit breakers”. Outside of a blogpost talking about seemingly different circuit breakers, AE could not find information about these circuit breakers in Chainlink’s official documentation.
External Oracle Interaction
If we look at Blizz Finance’s contracts on GitHub, there are a few issues that would be highlighted in an AE smart contract audit:
/// @notice Gets an asset price by address
/// @param asset The asset address
function getAssetPrice(address asset) public view override returns (uint256) {
IChainlinkAggregator source = assetsSources[asset];
return IChainlinkAggregator(source).latestAnswer().mul(1e10);
}The latestAnswer function can return zero if no answer is reached, and must be handled by the caller.
More importantly, Chainlink recommends using latestRoundData over latestAnswer to retrieve the price of an asset; latestRoundData will raise an error if the price feed does not have data to report and will not return zero. It also returns additional information that can be used to verify price data, such as the round ID and timestamp. Had Blizz placed checks on the roundID and timestamp returned from latestRoundData, they could have spotted that the LUNA/USD feed was no longer reporting up-to-date data.
Outside of checking timestamps, how can you identify a price feed is paused? To find the answer (and figure out these circuit breakers), we had to dig deeper into Chainlink’s data feeds.
Chainlink Oracles and Data Feeds
Reporting Prices
There are three layers of aggregation that occur when providing a price on data.chain.link:
- Price Data Aggregators take raw exchange data and create refined data sets for consumption
- Chainlink Node Operators take refined data sets and broadcast updates on-chain. (Done via node software written in Go, and an
AccessControlledOffchainAggregatorsmart contract) - The Oracle Network aggregates data from Node Operators, and can be queried by users (also called price consumers)
Chainlink provides additional details about data aggregation in their documentation.
Querying Prices
The flow of price data to a data feed consumer is roughly as follows:
- A consumer calls an exposed function on an
AggregatorProxycontract using theAggregatorV3Interface - The
AggregatorProxyforwards the call to anAccessControlledOffchainAggregatorcontract that receives updates from off-chain Chainlink nodes
A proxy contract is used in case Chainlink needs to swap out the underlying OffchainAggregator contract.
When do Prices Update?
Chainlink’s on-chain prices in an Aggregator contract update according to two trigger parameters: the deviation threshold, and the heartbeat. For example, the ETH/USD price feed will update when the off-chain price has moved more than 0.5%, or when the price hasn’t been updated in the last 3600 seconds.
The deviation threshold and heartbeat can differ between price feeds. Many price feeds have heartbeats longer than an hour.
Chainlink recommends pausing or switching to another feed if the answer is not updated within the feed’s heartbeat or “within time limits that you determine are acceptable for your application”.
Using these parameters, how quickly could a user identify a circuit breaker condition? Do they have to wait for a feed’s heartbeat, or even multiple heartbeats?
Relayer Deep-Dive
Aggregator contracts set several variables during initialization. The parameters of interest in this case are minAnswer and maxAnswer. If we peek at an aggregator contract’s constructor we can see their docstrings:
* @param _minAnswer lowest answer the median of a report is allowed to be
* @param _maxAnswer highest answer the median of a report is allowed to beThe minAnswer and maxAnswer variables are used to bounds-check prices reported by off-chain nodes when they call transmit to report a new asset price. In the case of the original LUNA/USD aggregator, they were set to the following values:
maxAnswer values on the LUNA/USD aggregator contractWith 8 decimals, this works out to a maxAnswer of 10000, and a minAnswer of .10.
Does that .10 look familiar?
The Circuit Breaker
Deployed contract code has the answers to our questions about circuit breakers. Note the examples linked are on Binance Smart Chain (BSC), and behavior will be consistent for other chains that had a LUNA/USD price feed since the contract code is the same.
If we look at the source for the AccessControlledOffchainAggregator contract, there is logic in the transmit function that checks the median reported value for extreme price swings:
{ // Check the report contents, and record the result
for (uint i = 0; i < r.observations.length - 1; i++) {
bool inOrder = r.observations[i] <= r.observations[i+1];
require(inOrder, "observations not sorted");
}int192 median = r.observations[r.observations.length/2];
require(minAnswer <= median && median <= maxAnswer, "median is out of min-max range");
If the median price is outside of minAnswer or maxAnswer, calls to transmit will revert. When the price of LUNA dipped below $0.10, we can see that nodes were trying to update the price, but their transmit calls were failing:
It makes sense that the last reported price of LUNA was reported was $0.10736, because it was the last time the median price was above minAnswer!
Some time after the circuit breaker tripped, Chainlink deployed a new Aggregator contract with a minPrice of 0.
Additional Safeguards
Knowing how these circuit breakers work, what can a protocol do to guard against unexpected oracle price behavior?
Call multiple oracles
Many protocols use more than one oracle to gather price data. Compound uses an individual ValidatorProxy contract for each token, which receives prices from Chainlink price feeds. The Chainlink price is then validated against the UniswapAnchoredView contract which checks the received price against a respective Uniswap v2 TWAP (time-weighted average price). Aave reaches out to Chainlink for prices, and uses a proprietary fallback oracle if they run into problems.
There are tradeoffs to consider when calling multiple oracles like additional code complexity and the fact that TWAP is slower to update than other price feeds, but multiple redundant sources of pricing data reduces the chance of oracle volatility affecting a protocol.
Off-chain monitoring
Off-chain monitoring can be used for introspection into a specific aggregator contract. In this case, protocols could have set up monitoring with a solution like OpenZeppelin Defender or Tenderly, to look for repeated failed transmit() calls with the message median is out of min-max range. Additional monitoring would be necessary to notify when an EACAggregatorProxy changes its underlying aggregator contract.
Multi-chain considerations
It’s important to keep in mind that Chainlink price feeds can behave differently on different chains. For example, there is a specific way to handle price feed outages on Arbitrum. Be sure to carefully consult documentation to ensure there are no edge cases for networks you intend to deploy on.
Above all, remember that oracle behavior can be strange with extreme price swings. Be very selective with what tokens your protocol chooses to handle.
Summary
While Chainlink provides a lot of information about interacting with their APIs, there is little documentation on their internal price circuit breakers. Components that can drastically impact end users should be documented; if it isn’t written down, it didn’t happen!
References and Resources
Thanks to harry.eth who spotted this circuit breaker behavior as it was unfolding:
https://twitter.com/sniko_/status/1524943313706565639
There are also some quality resources on safely interacting with external price oracles:
- Chainlink’s guide on Risk Mitigation
- OpenZeppelin’s guide on price oracle pitfalls
- samczsun’s “So you want to use a price oracle” blogpost
Looking for smart contract audit, or other decentralized technology security services? Check us out at arbitraryexecution.com.
We are always looking to add motivated researchers to our team. If this sounds like you, check out our careers page or drop us a line at info@arbitraryexecution.com.
