THORChain’s Layers of Security
Layers of security in place to make THORChain resilient to attacks and network uncertainty
Introduction
THORChain has almost reached its first milestone of maturity. The adolescent phase of the network, Chaosnet, is nearing its close. As the THORChain network prepares to enter its next phase of growth with the transition to Mainnet, it is first necessary to illustrate the growth and maturity of the network in the year since Chaosnet’s launch.
The priority for the team since the exploits has been a relentless focus on the security and stability of the network. These new features include blanket protections that make THORChain more resistant to attacks and tools to reduce the severity of a future attack should one occur. This article will show the steps and action taken since the exploits last year demonstrating THORChain’s focus on stability and security.
Exploit Investigation
Postmortems were conducted to fully understand what happened and why. Full Explanations of the THORChain exploits and response can be accessed:
As a result of the investigation, new security measures were announced THORChain Development Team Response — Hardening the Protocol which are expanded and added to in detail below.
Human Approach
Security measures are not always code-based. The below measures are in place to help protect and respond to network security issues as they occur. This requires the correct mix of expertise, decision making and vigilance.
Making Affected Users Whole
As the protocol was in Chaosnet, the Treasury decided to fully reimburse users for losses suffered during the exploit. Once the Chain restarted, Liquidity Providers and Node Operators were fully reimbursed approx $16M.
Bug Bounty Program
THORChain’s bug bounty program was formalized with ImmuneFi — the leading bug bounty platform for crypto native projects, paying large bounties for vulnerabilities. ImmuneFi serves as a neutral 3rd party to administer payments and verification of bounties on the network. Several bug bounties have been paid to date including a disclosure by Trail of Bits of a critical TSS vulnerability that had the potential to affect THORChain and other projects using TSS. The program will remain open and undergo periodic review.
External Code Audits
Audits of the THORChain code have been completed by some of the leading blockchain security teams including Halborn Security and Trail of Bits. There were no unresolved critical security vulnerabilities at the time of the issuance of the reports. The full report for every audit on THORChain can be found here.
Team Focus on Security
The THORChain core team, with assistance from Nine Realms, have overhauled its security posture. While security and stability were always a priority, a strong and persistent focus on security and network stability is now baked within the network operations and feature addition processes, ensuring it is always the primary focus. This focus has tempered feature release speed and has ensured every step is a methodical and deliberate process allowing for sufficient testing before releasing to Chaosnet.
THORChain inherently requires a large dependency footprint, and supply chain attacks are one of the most likely sources of vulnerability. The Nine Realms and THORSec teams continue to mitigate these risks by reducing dependencies, pinning version hashes, and working towards a final goal of distributed reproducible builds from source for the entire set of THORNode daemons.
ThorSec “Red Team”
Nine Realms was introduced and has overseen the creation of a 24/7 always-on security team to work offensively and defensively on the security of the network. This team is composed of world-class whitehat security experts who have uncovered major exploits of other networks in the past. The responsibilities of ThorSec include:
- Review and approval of Pull Requests to THORChain’s open-source codebase.
- Provide 24/7 Monitoring of the protocol health
- Responding to issues, events and incidents
- Creating run books and procedures for network updates, chain additions, hard forks
- Monitoring for connected chain vulnerabilities, upgrades and hardforks that could impact the network
While great strides have occurred, it is understood this will always be a continual work in progress.
White Hat Disclosures
A detailed unstructured review of THORChain’s THORNode and Bifrost code base was conducted by a seasoned security researcher who has responsibly discussed 10 vulnerabilities of varying severity. All vulnerability disclosures can be reviewed at https://gist.github.com/HildisviniOttar. Each vulnerability has been patched.
Introduction of Stagenet
Stagenet was introduced which is a staging environment that is a clone of THORChain using real assets and closely mimics Chaosnet. Stagenet allows rapid iteration through wargaming/testing of new features and the addition of new chains. All new features are tested in Stagenet first with lucrative bounties on offer for issues found, before being released to Chaonset.
Hardening the Code
THORChain operates alongside the general principles and ethos of Bitcoin, although its surface area and security model varies greatly. THORChain is completely open-source and transparent to encourage community development, responsible disclosures, allow open audits, and welcome constant improvement of the protocol. While being open-source, conducting audits and reviews gives confidence, they do not ensure there are no unknown vulnerabilities or that none will be unintentionally introduced in future updates. With this understanding, additional protocol-wide security measures to protect user funds and the network have been added as blanket protections to either stop or greatly reduce the risk to funds in the event of future exploits.
Additionally, these safeguards have been implemented to encourage bad actors to report their potential findings through the ImmuneFi bug bounty program rather than carrying out an attack if a vector is found, as it will likely be more profitable.
The following protections are currently coded in and supplement the human approach.
Mimir Control to Nodes
Mimir is a feature that allows the constants and parameters of THORChain to be edited. THORChain is governed by the nodes that have posted bonds to validate transactions on the network. Mimir gives governance powers through node consensus (>66%) to change constants such as Minimum Bond, Churn Speed, Incentive Pendulum, Reserve Emissions, and more. Active Nodes can vote to change network parameters to ensure the optimal performance and safety of the network. More information about Mimir can be found here.
Admin Mimir access will be depreciated during Planned Obsolescence when control is fully given to the Nodes.
Automatic Security Flagging
Security events have been added within the code which automatically emits events when certain conditions are met such as node slashing for unauthorized transactions, attempted double-spending, or very large withdraws. Emitted events are sent to a THORSec Discord monitoring channel for immediate review. These emitted on-chain events can also be seen on Midgard. Relevant PR.
Granular Halting Controls via Node Mimir
THORChain requires different halt controls due to the complexity of the THORNode software. Issues may arise that affect a connected chain of THORChain and not others, e.g. a chain fork, node syncing issue, or an exploit. Improved halt controls give Node Operators more control over the safe operation of the network as well as the responsibility to halt trading if an issue arises through Mimir.
Nodes are given the ability through Mimir to:
Halt Signing for a Specific Chain
- Mimir Setting: ‘Halt{Chain}Signing’
- Effect: All outbounds are queued (e.g. withdraws and output of swaps)
- Use Case: Chain-specific issue (e.g. external hardfork) or large Node signing issues.
Halt Trading for a Specific Chain
- Mimir Setting: `Halt{Chain}Trading`
- Effect: Pauses liquidity additions/withdrawals and swaps, on a specific chain (eg. ETH only).
Use case: Chain-specific issue, solvency issue detected, or unauthorized transaction detected.
Halt a Specific Chain
- Mimir Setting: Halt{Chain}Chain
- Effect: Pauses all observations on that chain. Nodes will not be synced to the tip.
- Use cases: Introducing a new chain, major chain-specific issue.
Halt Trading Globally (Network Wide)
- Mimir Setting: `HaltTrading`
- Effect: Stops all trading and observations for every connected chain. The THORChain blockchain will continue and native RUNE transactions will be processed.
- Use Case: Emergency Control, Node triggered trading halt.
Relevant PR. More controls and information can be found here.
Automatic Solvency Checker
The solvency checker of the network’s primary vault was introduced to catch issues early or prevent them before they happened. Detected issues can trigger an automatic trading halt on a specific chain. There are two types of automatic solvency checks:
- Reactive: The network continually compares the assets the network believes it has in its primary vault (Asgard Vault) to what assets are actually stored in the network’s chain wallets. For every connected chain, each node compares the stored Asgard vault value to each chain Asgard Wallet amount. If the Asgard Vault value is more than the wallet value by 1%, the node reports insolvency for that chain. Relevant PR.
- Proactive: This mode is more powerful and is intended to catch insolvencies even before they appear. When a node attempts to sign a txOut, it will do a calculation to check if by executing the txOut the vault becomes insolvent. If so, then it refuses to sign and reports insolvency. Relevant PR.
Automatic Trading Halting
If more than 66% of nodes, report an insolvency with a chain (reactive or proactive), trading for all assets on that chain is automatically halted (Halt{Chain}Trading). The halt will also emit a security event to ThorSec.
Why 1% difference allowance?
THORChain builds an awareness of balances purely by adding the incoming funds, then subtracting the outgoings. Thus the expected balance is the aggregate of the ins and outs. Divergences can occur when actual on-chain balances start to diverge. For gas assets (assets used to pay gas), small divergences can be intermittent, but normally less than +/- 1%.
Secondary Vault Security
Yggdrasil Vaults, the secondary vaults that hold approx 25% of the network funds, are secured by the Node Operators’ bonds and the threat of their bond being slashed 1.5x of any funds they mishandle (e.g steal or attempt to double spend).
Node Operator Triggered Halts
Node Operators have the ability to halt trading within THORChain if they detect malicious activity. This is in addition to automatic chain-specific halting functionality. The ability has been made with the potential for abuse in mind.
A single node can pause trading (HaltTrading) for up to an hour (720 blocks). Any additional node that calls for a halt will add 720 blocks to the halt timer. Any node that calls for the resumption of trading removes 720 blocks from the timer. The halt functions can only be called once by each active node per churn cycle (3 days). This gives the network the ability to respond to a malicious threat without giving any node unilateral control. Relevant PR.
Unauthorized Transaction Detection
This blanket protection will automatically pause THORChain on a specific chain (Halt{Chain}Trading) and stop Yggdrasill vault funding if an unauthorised transaction is detected. This will emit a security event and be sent to THORSec. The halt will allow Node Operators, THORSec, and the development team to investigate and react immediately following an unauthorised transaction and issue a fix and restore service as quickly as possible. Relevant PR.
Outbound Transaction Throttling
To prevent large amounts of funds from leaving the network in an instant, large outbound transactions are throttled from immediately leaving the network. Each block has an outbound value limit and each outbound transaction has a maximum time limit that it can be processed. This has three effects:
- Each outbound transaction to compete for the next outbound block, else, it will be processed in the following block, effectively throttling the total outbound capacity of the network. This is independent of conf-counting.
- Large outbounds to spread across multiple blocks, up to 720 blocks (approx one hour).
- Ensures one large outbound request of $1,000,000 is handled the same as one million $1 outbound requests.
This feature is controlled by several Mimir values that can be changed by Node Operators as required. Relevant PR.
This serves as a defensive layer buying time and allowing a vigilant node operator to pause trading or automatically halt checks to engage before the large malicious outbound transaction is irreversibly executed and funds are lost. While the feature negatively impacts the user experience of THORChain, but is necessary in ensuring the protection of liquidity provider funds.
The feature is designed to grow as the network grows and is controlled by several Mimir values that can be changed by Node Operators as required. Additionally, the feature is designed to promote bug disclosure instead of directly attacking the network as a bug disclosure is likely more profitable.
The THORSwap team has created a dashboard to view delayed outbound transfers which is available here.
Live Monitoring Tools
Live monitoring tools have been are in place to assist in the safe operation of THORChain. Bots have been created to alert when abnormal activity occurs on the network which is posted on Discord and Telegram channels. This gives node operators the required tools to investigate when unusual activity occurs and to take the appropriate actions. These tools are in addition to the security event alerting previously described.
Discord Monitoring
A dedicated THORChain Developer Discord channel called ‘chaosnet-info’ receives bond slashes, node Mimir voting, large additions or withdraws, and other chain metrics.
Node Messaging
A bot was added to THORNode to allow node operators to communicate publicly and transparently in the THORChain Developer Discord while maintaining anonymity. This also creates an anonymous but open forum to discuss node and network operations.
Telegram Monitoring
Telegram monitoring bot is also an excellent tool to monitor unusual activity on the THORChain network including node mimir votes, network pauses, large liquidity deposits, large swaps, and other critical network information. The Telegram bot can be accessed here.
Maximum Number of Nodes To Churn Out for Low Version*
This Mimir parameter titled “MAXNODETOCHURNOUTFORLOWVERSION” allows changes to the number of nodes that can be churned out for not running the latest version of THORNode. Previously, this limit was globally set to “1” which could leave the network in a vulnerable state if multiple nodes were not updating. If there were a critical update and more than one node did not update to the most recent version, it would take multiple churns to remove nodes that have not yet updated. Relevant PR.
*Currently not yet enabled, being voted on by Node Operators via Node Mimir voting.
Cloud Provider Distribution Enforcement*
This new network parameter sets the maximum percentage of nodes that can be hosted on one cloud provider service. Node infrastructure is costly and advanced sysadmin skills are required to host a bare metal node. Therefore, many node operators choose to run their nodes on cloud provider software such as AWS, Azure, DigitalOcean, or Google Cloud Services. This presents a major risk due to insufficient decentralization.
For example, with a CloudProviderLimit of 3333, no more than 33.3% of nodes can be hosted on the same cloud provider. This ensures that even in the event of complete failure of a cloud provider, THORChain will continue to produce blocks and achieve consensus. Relevant PR.
*Currently not yet enabled, being voted on by Node Operators via Node Mimir voting.
Summary
The THORChain network has matured significantly since the launch of Chaosnet and added many layers of security. While there can never be a guarantee regarding security, the measures outlined in this article should give confidence to users that the impact of any attack will be greatly reduced and hackers are more incentives to disclose bugs than attack the network.
While the specifics of the security measures taken by the network may change, the focus on security remains the highest priority of the network and its participants.
Today, THORChain is more impervious to attack, and is able to react quickly to save funds, THORChain is ready for the next phase of growth.
THORChain Community
To keep up to date with the project, please monitor community channels, particularly Telegram, Discord and Twitter:
- Twitter: https://twitter.com/thorchain_org
- Telegram Community: https://t.me/thorchain_org
- Telegram Announcements: https://t.me/thorchain
- Discord: https://discord.gg/7RRmc35UEG
- Reddit: https://reddit.com/r/thorchain
- Github: https://github.com/thorchain
- Medium: https://medium.com/thorchain
