Kusama and Karura Governance Coordinate to Neutralize Would-be Exploit

Karura is built on Kusama, Polkadot’s wild cousin, whose creation was intended to serve as an experimental hotbed of mostly unaudited, pre-production code for parachain teams to implement, battle test, and strengthen before it ever makes it to Polkadot. Kusama’s blockchain development framework called Substrate is also relatively new and constantly changing. The framework is exponentially more powerful than previous blockchain development methods, and with this power, comes risk.

Karura’s engineering team is the top Github contributor to the Polkadot and Substrate codebase outside of Parity Technologies. The team has been pushing Kusama and Substrate features to their limits, as intended, including being the only parachain leveraging cross-consensus messaging (XCM) reserve asset transfers between Karura and the Kusama Relay Chain.

Kusama has recently lived up to its experimental nature in a recently neutralized exploit due to a bug caused by a previously undiscovered version combination between cross-consensus messaging (XCM) code on Kusama and a different XCM version on Karura. This vulnerability has since been remediated, resolved, and fixed with network upgrades across all parachains. Overall, the result is a shining example of effective mitigation strategies in the code and effective on-chain governance coordination between multiple networks. Additionally, the Kusama ecosystem has benefited from this identified bug and versioning issue since it has resulted in several lessons to implement as Kusama and Polkadot continue to grow.

What happened

In ongoing monitoring efforts on October 12th, 2021, the Karura team found a mismatch between the total amount of KSM on the Karura tokens pallet in comparison to the total KSM in the Kusama parachain account on the Kusama Relay Chain. The discrepancy was 11,000 KSM less than it should have been in the Karura parachain account on the Relay Chain. We then identified suspicious transactions using Kusama’s cross-consensus messaging code:

• Transaction 1
• Transaction 2
• Transaction 3

Our remedial action

After discovering the discrepancy, we notified the exchange from which the attacker sent KSM funds to the account used for the exploit. Karura’s governance body then voted to use a special ‘transaction pause’ pallet to halt raw XCM transactions on the network in order to give time for Karura and Kusama governance to take remedial action. This contingency pallet was built by the Karura team to prevent events like this from occurring, allowing governance to swiftly neutralize exploits.

Karura governance then worked with Kusama’s governance body to successfully recover the KSM funds. Kusama then proposed motion 373 to transfer 9,999.99 KSM back to the account, which was fast-tracked by the technical committee. The remaining 1,000 KSM was burned by the attacker and a proposal will soon be made to the Kusama council to re-mint this KSM back into the Karura sovereign account. The result of these remediation efforts will be no loss in user funds. With the funds recovered and the proposal to re-mint forthcoming, Karura and Kusama were then able to upgrade the XCM code to XCM v2, neutralizing the vulnerability and establishing a new framework to improve compatibility amongst multiple parachains’ implementations of XCM.

How it happened

This happened due to the compounding of multiple factors across Kusama, XCM, and Karura. The Statemine chain was updated by the Kusama/Statemine team, and Karura implemented the same XCM code used by Statemine as a reference.

Initially Kusama was running runtime 9090, Karura was running runtime 1011, and XCM transfers worked perfectly well. Then Kusama upgraded to runtime 9100, which should have backward compatibility, but did have one breaking change that broke the cross-chain transfer ability of KSM from Karura to Kusama. The Karura team notified the Kusama team about this issue and started working on Karura runtime 1013 to fix this issue. One unintended side effect of this upgrade was that the exploit was possible with this combination of XCM versions on Kusama and Karura. This exploit was only possible when Karura was using XCM v0 and Kusama was using XCM v1.

This was a privileged escalation bug that allowed anyone to craft and execute an XCM message on Karura in such way that would result in the Karura parachain sending an XCM message to Kusama (that’s ok) with an origin of the Karura parachain account (not ok) in order to execute a reserve withdraw and deposit KSM to exploiter’s account. A few attempts were made from the exploiter and results 1 KSM, 1,000 KSM, and 10,000 KSM transferred out of the Karura parachain account. 1 KSM and 1,000 KSM were burned from Kusama due to the usage of XCM, and only 10,000 KSM were successfully transferred out from the Karura parachain account on Kusama.

In a few hours, Kusama upgraded to runtime 9110, which upgraded the network to XCM v2 fixing the bug and ensuring that the exploit is no longer possible. The Karura runtime 1014 included a patch to fix the XCM compatibility issue as well as prevent this exploit.

Below is the summary of releases and XCM versions:

Karura

• Karura 1.4.3 / Karura runtime 1011 — XCM v0
• Karura 1.4.4 / Karura runtime 1013 — XCM v0 with compatibility improvement for XCM v1 & v2
• Karura 1.5.1 / Karura runtime 1014, Latest release — XCM v1
• The upcoming version of Karura (Not yet released), XCM v2

Kusama

• Polkadot v0.9.9 / Kusama runtime 9090, XCM v0
• Polkadot v0.9.10 / Kusama runtime 9100, XCM v1
• Polkadot v0.9.11 / Kusama runtime 9110, XCM v2

Our History of Code Contributions

The Karura team prides itself on working for the betterment of the Kusama, Polkadot, and Substrate ecosystem, having solved a multitude of bugs and contributed several pieces of open-source code to improve the ecosystem over the past 2+ years:

1. UMP Weight bug: It was believed to be safe to reduce UMP weight, but after more investigation of Acala/Karura team, we found out it is unsafe to reduce it and there was a critical bug in XCM that if multiple parachains are sending XCM messages, some messages will be dropped and result in loss of funds.

• The referendum: https://kusama.polkassembly.io/referendum/134
• The bug report: https://github.com/paritytech/polkadot/issues/3782

2. XCM bug: During cross-chain testing between an internal testnet of Karura and Kusama, we identified that cross-chain transfers no longer work on the next release version of Kusama and reported this issue to Parity. It was later identified that this was a compatibility issue between XCM v1 and v0 that would make any kind of cross-chain transfer on Kusama stop working. https://github.com/paritytech/polkadot/commit/18912e40e22120c792f48ea2237d776a31e83eb2

3. XCM bug: Acala/Karura team identified that cross-chain transfer stops working after Kusama upgraded to use XCM v1. This affected both Karura and Statemine. A motion is proposed to force set default XCM version to v0 to ensure the compatibility. The referendum: https://kusama.polkassembly.io/referendum/138

4. XCM simulator tool: Shaun from the Acala/Karura team has developed an open XCM simulator that allows anyone to test cross-chain transfers and other more complicated cross-chain integrations between multiple runtimes. This has been later contributed and merged into the Polkadot Github repo. This makes it possible to write tests for xcm related code. https://github.com/paritytech/polkadot/pull/3538

5. Other important contributions from our team:

• https://github.com/paritytech/polkadot/pulls?q=is%3Apr+author%3A%22shaunxw+%22+is%3Aclosed
• https://github.com/paritytech/polkadot/pulls?q=is%3Apr+author%3A%22xlc+%22+is%3Aclosed+
• https://github.com/paritytech/cumulus/pulls?q=is%3Apr+is%3Aclosed+author%3Ashaunxw+

What we learned

Begin monitoring balance checks across parachains: Karura is already building tools to help development, testing, and monitoring across parachain networks. It’s our intention to coordinate with the ecosystem on establishing a clearinghouse to monitor amongst all sovereign blockchains connected to Kusama. There is also a need for a more effective dashboard business logic and asset monitoring for ensuring network security.

Begin communicating guardrails and caution to parachains: The Kusama team can improve the communication of critical pieces of code by providing parachain teams with words of caution when applicable. Kusama and Polkadot have communicated that they will now use “locked files” and “locked lines” for highly sensitive pieces of code that require enhanced review by multiple experts in order to be changed in the repository.

Continue using and improving Karura’s contingency pallet mechanism: Karura will continue to improve contingency plans such as our “transaction pause” pallet built specifically for situations like this where Karura’s governance bodies can vote to halt transactions of bad actors.

Test collectively across different versions of Kusama for compatibility: We aim to contribute to a collective effort across parachains and Kusama to compare and verify testing results across different versions of the Kusama code. We should all ensure the new Rococo Revamp tool can help us achieve this goal.

Continue completing audits with top audit firms: Any non-trivial code on Acala goes through third-party security audits, completed by firms including SR Labs (also used by Polkadot), Trail of Bits, and Slowmist. Additionally, Acala is committed to ongoing security audits and has contracted these leading firms to perform periodic auditing to ensure successful deployments and a sound code base.

Initiate a Vulnerability Advisory Program for the ecosystem: We have begun taking steps to form a Vulnerability Advisory Program to help notify all connected teams and ensure smooth transition to new code versions to mitigate vulnerabilities.

Takeaways

This event has underscored the importance of having a “canary network” with both Karura and Kusama where we can continue to push this technology to its limits in order to innovate as well as find vulnerabilities to fix before the code makes it to a higher-risk environment on Polkadot. We were also pleased with the ability to move swiftly to enable the special pallet via Karura governance, then work with the Kusama governance to reverse the transaction that took advantage of Kusama’s XCM bug. This was a valuable learning experience in how to navigate governance decisions quickly and effectively to protect our networks. All in all, our team’s contributions of identifying a new vulnerability, highlighting needs to improve communications and testing strategies, and improving monitoring across the parachains are only the beginning, and we’re committed to building on any efforts to make the Kusama and Polkadot development ecosystem more resilient.