Postmortem Report | Nov 15, 2022

Published in

Findora Foundation

4 min readNov 22, 2022

This postmortem report is written for and provided to the community to provide transparency into a recent outage. This article will describe the event, its root cause, and its resolution while highlighting how improvements will be made in an effort to prevent a recurrence of the issue.

Event Date

Tuesday, November 15, 2022

Impact & Duration

Mainnet consensus halted
Duration: 4.5 Hours

Overview

The Discreet Labs DevOps team attempted to coordinate with a set of Findora community validators for a pilot upgrade on the Findora Mainnet network. During this attempt, the Mainnet network experienced an unexpected outage. Engineers identified the issue, immediately began troubleshooting efforts, and worked with the affected validators to revert the nodes to their previous version. More community validators joined the effort to restore their nodes.

What Occurred

At approximately 5 PM, Nov 15, 2022 (PST), the Findora Mainnet’s consensus stopped for 4.5 hours. The incident was triggered, not caused, by several transactions sent from a newer beta version of the Findora wallet after DevOps coordinated with a set of validators during a pilot upgrade with the v0.3.3X Mainnet update.

The event was detected proactively by the event monitoring system. Discreet Labs began troubleshooting efforts within 10 minutes of the notification.

Root Cause(s)

The v0.3.3X update includes a new transaction parameter in the transaction body. The transaction hash calculation logic of the node is as follows:

Deserialization -> Signature Verification -> Transaction Execution -> Serialization -> Compute Transaction Hash

Since the new parameter did not exist in previous versions, nodes running v0.3.27 ignored the parameter when calculating the transaction hash. In version 0.3.3X, however, this field is involved in the transaction hash calculation, thus causing the transaction hashes to be inconsistent.

Resolution

The new transaction parameter will be ignored for hash calculation in v0.3.3X but will retain its role in signature verification. This update to release v0.3.3X will ensure transaction hashes remain backward compatible with v0.3.2X nodes.

Moving Forward

The Discreet Labs team has documented several lessons learned and preventive measures to take moving forward. The items below reflect the methods teams will use collaboratively to minimize risk and optimize communication.

Additional Testing via Mainnet-Mock

Upgrades with impact potential will undergo additional rounds of testing on a transient network called “Mainnet-Mock”. This network will be an identical copy of Mainnet, including its data and validators, acting as a mirror of Findora’s production network.
Mainnet-Mock will be created and deconstructed as needed. As a mirror copy of Mainnet, testing against the Mainnet-Mock network will require the assistance and close communication of Findora community validators to reach > 66.67% consensus.
Upgrades that carry potential impact will be validated on the Mainnet-Mock network before being classified as “Mainnet-Ready”.

Change Management Process Improvements

Changes that can impact production will go through an improved change management process. These process improvements will ensure that all the appropriate stakeholders are in-sync with the engineering change task.

Discreet Labs will continue to ensure that partners and CEX contacts are notified of upcoming changes using the schedule above. Technical support will continue to be provided when needed.
The details of changes that can impact production will be communicated extensively with the community well in advance of the scheduled change date. Discreet Labs will share information two weeks before the change, one week prior, three days prior, one day prior, the day of the change, and the day after.
Change information will be shared across social media, pinned to the official Discord and Telegram channels and Reddit as announcement posts. Discreet Labs will communicate to the validator community on validator-specific channels across Discord and Telegram.
Mainnet-Mock testing will be conducted as part of the change management process. See above for information on Mainnet-Mock.

Special thanks to the validator community who worked together in restoring their nodes in a collaborative manner, reflecting the positive values the Findora community fosters and embraces. Your decision to help decentralize and secure the Findora network is greatly valued.

We look forward to upgrading Mainnet to v0.3.3X together.

About Findora

Findora builds privacy through advanced zero-knowledge proof cryptography. An innovative layer-1, it combines a native UTXO ledger optimized for privacy with an EVM extension for programmability and interoperability. Developers can leverage either model as they build dApps with auditable privacy.

We appreciate our developers and would love to onboard you to the Findora ecosystem. Please reach out, and join our social channels for more.