The ixo blockchain went offline on December 29th 2022 due to an IBC upgrade error that resulted in an application panic that halted block processing. We want to briefly share some the lessons we learned and to acknowledge the incredible support of the Cosmos core developer community. Also, our sincere apologies to anyone who was adversely affected by this infrastructure downtime.
Resolving the issue required an extensive process of diagnosing and testing for the root cause, which was a bit of a search for a proverbial needle in the haystack, that took longer to resolve than what we would have liked because of the New Year holiday period. The chain was re-initialised with a restart on January 12, 2023.
IBC upgrades
The Mundo upgrade of the ixo main-net on December 8th 2022 included updates in the versions of the Cosmos SDK and IBC, reset the block number, and changed the chain-id.
During this upgrade process all IBC channels became frozen as the upgrade team had not followed the UpgradeProposals process for existing IBC Clients before implementing the upgrade. This resulted in all existing IBC Clients expiring on December 25th. When users attempted to perform IBC transfers, these transactions could not be processed correctly and this disrupted the consensus status of the chain.
With the chain restart, new governance proposals have been submitted on the ixo chain and counterparty networks, to reinstate the IBC channels.
When upgrading
The rapid state of development of the Cosmos SDK and related protocols, such as IBC, makes the process of planning and successfully implementing chain upgrades quite challenging and we would advocate for calling in peer support and thorough reviews prior to going live with main-net upgrades, even if everything appears to be working in your test-net environment.
To assist Cosmos developers in future, a Github action is being developed that can be easily imported to run automated upgrade tests for chain.
The timing of this incident was bad, as most people were unavailable over the holiday period and we were not able to fully coordinate a response until the 2nd of January, with the solution being implemented on the 9th of January. Perhaps another lesson to be learned is not to perform network upgrades too close to the holidays!
Go Cosmos!
This experience has reinforced our belief in the spirit and competence of the Cosmos community. We are hugely grateful to the Cosmos core developers and collaborators who stepped in to assist our team with diagnosing this problem, and to the ixo validators for working with us to implement the necessary fixes.
We want to specifically acknowledge Charly from the IBC team at interchain.io, Adi from Informal Systems, Ankur from cheqd network, Jacob from Notional, and Simo at Active Nodes. Alwyn Van Wyk from the core ixo team coordinated the response with great personal commitment and sacrificed precious family time over the holiday period.
The ixo Validators did a great job of coordinating the steps fix the chain.
It takes a community to run a blockchain
Upgrading and maintaining public blockchain infrastructure requires a collective effort and through this experience we feel even more connected to this great Cosmos community!
