Incident Overview
We regret to inform our community that we have experienced a failure in the Plutus-Chain-Index module used by the Cardano Open Oracle Protocol (COOP) in the current version of the Orcfax network. After several recovery attempts and careful deliberation on network impact, we decided to temporarily suspend the publication of our ADA/USD feed as of May 15th. No feed data has been published since. We promptly notified Cerra, the one project using the current version of the mainnet feed in production, allowing them to activate their backup solution. In the meantime, we have deployed the Orcfax v1 protocol upgrade to testnet. This upgrade eliminates the technical debt dependence on IOG’s deprecated Plutus-Chain-Index and introduces many other network improvements.
Background and Current Situation
As soon as we released the first version of COOP in mid-2023, IOG discontinued support for Plutus Application Backend and Plutus-Chain-Index, two critical components in the COOP stack. We were aware of the discontinued support and have been working with our technical consultants MLabs over the past 7 months to replace these components. We had expected this upgrade to be completed sooner and to have moved past the Plutus-Chain-Index dependency at this point. Unfortunately, this tool failed before a replacement was delivered to us.
We launched the first version of our ADA/USD feed using COOP in late 2023. We incorporated input from early feed integrators and our own testing to identify a number of protocol upgrades. We quickly identified the need for a less verbose and more economical datum value, while also recognizing this would require a one-time, backwards-incompatible feed upgrade.
The stability and predictability of our feed datum format is key for our data consumers so we contacted the projects using (and planning to use) the first protocol version in early 2024 to inform them to stand-by for this datum format upgrade. We have also developed, and have been routinely assessing, our own on-chain feed consumer monitoring feature, which allows our team to identify Orcfax feed integrators. Together, these processes and tools gave us the assurance that no production protocols would be affected by this decision and temporary service disruption.
Going forward
Like other Cardano projects, Orcfax fell victim to outdated, unsupported infrastructure components and the need for more mature ecosystem-wide tooling. As a small startup project, Orcfax does not have the resources or motivation to fix components abandoned by Cardano core developers. Therefore, we decided to focus our efforts on the new, go-forward version of the Orcfax protocol rather than fixing something that is already deprecated.
Fortunately, the Cardano infrastructure ecosystem is maturing, as witnessed by three of our developers who participated in the recent Cardano’s BuidlerFest. Our focus now is on collaborating with other Cardano community initiatives to integrate more lightweight open-source components into the Orcfax protocol.
Going forward, our architecture will only include forward compatible upgrades, which will be rare and communicated months in advance. Our upgraded datum format will also allow feed consumers to verify data validity on-chain without requiring any third-party monitoring and notifications.
We are currently testing our upgraded protocol on Cardano Preview and Preprod networks. A critical piece of this work is establishing the scaffolding to ensure Orcfax’s many long-running processes can be monitored and have appropriate error-recovery mechanisms to circumvent and manage any future incidents.
We have also begun to onboard our community participants into our Incentivized Testnet (ITN). This will allow us to test and harden the components which will run the fully decentralized version of the Orcfax network. Of course, supporting automated disaster recovery and long-running processes will be a key focus in this process.
Other enhancements that will be introduced in the upcoming Orcfax protocol upgrade include:
- Improved datum schema: lower transaction costs and easier dApp integration
- Multiple feeds in one transaction: lowering transaction costs by up to 75%
- Accurate CNT feeds: aggregating all transactions, per CNT across DEX pools to deliver accurate and timely token prices on Cardano.
- Enhanced Explorer: adding more feeds and improved usability to our Orcfax Explorer
- Community supported components: publish and contribute to community-based Cardano infrastructure tooling
Conclusion
The Orcfax oracle stopped publication of its mainnet ADA/USD feed due to a critical failure of the Plutus-Chain-Index component in the Cardano Open Oracle Protocol (COOP) stack. This is an anomalous event that did not have any financial consequences on Cardano mainnet. The Orcfax team was already working on multiple remediations to prevent a similar occurrence in the upgraded version of its network. Unfortunately these were not completed in time before the failure of this deprecated component. We made the cost-benefit decision to abandon further recovery work and focus instead on accelerating our protocol upgrade. While unfortunate, we consider this as the best course of action under the current circumstances.
We hope to deliver our protocol upgrades very soon and demonstrate our approach to delivering authentic and accurate real-world data to Cardano dApps in the most robust, economical, and reliable way possible.
Thank you for your continued interest and support.
Sincerely,
The Orcfax Team
