The Kava mainnet launch was scheduled to occur Nov 5th, 2019 at 14:00 UTC. The proposed genesis file was released at approximately Nov 4th, 2019 at 14:00 UTC. Validators coordinated to verify the correctness of the genesis file, and a minor issue with the vesting terms of a few accounts was uncovered. A patch was applied to the genesis file around Nov 5th at 08:00 UTC. Communication was made through social media to make validators aware of the updated genesis file and then need to restart their node if they had already downloaded the previous genesis file.
At 14:00 UTC the network began to come online. The network was unable to reach a quorum, and we soon heard from validators running v0.3.6 of the
tmkms signing system that their nodes were reporting attempted double signs. Due to the seriousness of double signing, validators using
tmkms felt uncertain about resetting their nodes to try and start the chain. It was believed a quorum could still be achieved by nodes not using
tmkms and the issue would not be a problem after the network was live. The validator community worked to locate and assist validators who were currently not on the network to come online.
At around 16:00 UTC a block was produced by a quorum of ~75% of nodes. However, when the second block was proposed, it was reported by many validators that a fork had occurred which was causing some nodes to report
Wrong Block.Header.AppHash for the second block. The fork was believed to have been caused by a node or nodes that were not running the latest version of the genesis file. A chain deadlock ensued for the next ~30 minutes in which the fork was not resolved in 31 consensus rounds. During that time validators began discussing the potential need for a relaunch due to the unresolved fork and weather or not the re-launch should occur immediately, or if time should be taken to re-coordinate and potentially resolve the issue with
tmkms in order to make bringing the network up easier.
During this discussion, the fork was resolved and the chain began making regular blocks as validators delegated their stake on the winning chain. We were immediately contacted by validators unsure why their node had not come online, and unsure if the fork contained the correct genesis state. Users of the
tmkms were concerned about what the consensus state of the chain showed, since if it was inconsistent with their node it could cause a double sign. During this time I noticed that a double sign had occurred shortly after the fork resolved. Deciding that there had been a coordination failure and many validators were unsure of how to proceed, I immediately recommended we abandon the launch so we could conduct a proper post mortem, fix any security risks, and coordinate a smoother launch.
In the immediate aftermath, it is clear that releasing the genesis file 24 hours before launch, and making an update 6 hours before launch, greatly increased the risk of launch failure. Further, although we conducted a successful testnet launch with a code-frozen version of
Kava prior to mainnet launch, many validators did not participate using production systems, which left us vulnerable to risks particular to those systems. In the coming days, we will conduct a post mortem of what occurred and take steps to address and correct these issues going forward.