If you’re connected to the Cosmos community, you likely would have seen various alerts and messages going around about an expedited update requiring a hard fork.
The big take-home: all funds are safe and no ATOMs were at risk, and the ICF is expected to publish a full post mortem of the issue, the exploit, and the response in the next 6 – 9 business days.
June 17 2019 update: here is the post-mortem issued by Jessy Irwin.
Quick summary
The issue? On May 28, the Tendermint team was informed of a critical security vulnerability in CosmosSDK. They deemed the vulnerability to be of high severity, since it posed a risk to the security model of the Cosmos’ proof of stake (PoS) incentive mechanism.
In response, AiB and ICF members quickly wrote the patch (a small change) and implemented it. They then co-ordinated a disclosure process, involving the projects building on CosmosSDK and the Cosmos validator set, to expedite a quick hub update in order to prevent exploits in the wild. The teams posted publicly on the Cosmos forum and Twitter, word spread quickly, and then the validators used governance mechanism to signal preparedness to execute the hard fork.
Validators decided to use the Cosmos governance mechanism to signal that we had upgraded our nodes and were ready for the impending hard fork at block height 482100.
What resulted was a governance quorum of 84% (with 100% yes votes) and it happened within six hours. A vast majority of validators participated, and the Cosmos Hub network forked seamlessly just before 9pm EST.
The Tendermint team started a livestream around 200 blocks prior to upgrade, which you can check out here: https://www.youtube.com/watch?v=ValbS5rTdgc&feature=youtu.be
Going forward, we can expect a few things. The current chain will share the history of the pre-fork chain, remaining essentially the same, and thus it will continue to be referred to as ‘Cosmos Hub 2.’ We can also expect the All in Bits and Interchain Foundation teams to provide technical details about the vulnerability in the coming week.
Looking back
The Tendermint team began quietly reaching out to Figment Networks and the other validators on the evening of May 29. They worked with us overnight to ensure 1) that we understood the software upgrade and the reason for the upgrade, 2) that we would be available to respond during the network transition at block height 482100 and 3) that we were prepared to run the updated CosmosSDK software.
For Figment Networks, ensuring a seamless transition meant sleepless perseverance and perfect uptime. We were impressed by the effectiveness of the Tendermint team. It was equally impressive to see the the strength of the validator community, which proved itself very capable of getting the word out and co-ordinating this upgrade.
Lessons learned
In order to keep stakeholders safe, the timeline for this upgrade had to be fast, and at the cost of uptime for several of the validators. Why so fast? The exploit becomes more likely to be abused once announced. After the software update was released, it was effectively an openly-published exploit. Anyone with the right knowledge may have looked at the Github pull request to figure out and abuse the exploit.
During this expedited upgrade process, we were fortunate not to have lost any uptime, though we can expect that nearly every validator will experience downtime eventually. Regardless, we know that the network itself is resilient enough manage a hard fork in under 48 hours. The Cosmos community showed its strength and its dedication to building and running this chain.
This upgrade will serve as a low-risk drill to prepare us for more severe security issues in the future. It challenges validators to quicken our response times with proper monitoring, and to plug in to the many streams of the Cosmos community to stay informed.
We’re grateful for the efforts of the entire community, and look forward to strengthening our network with every experience.
Update: we have disclosed our Cosmos validator operations here — check us out!
Hopefully you found this useful. Feedback is always welcome!
Figment Networks is on Twitter | Medium | Discord | World Wide Web 🌐
