Pending Validator Deposits and the Migration Process

EDIT — July 5th, 2023: Most Genesis Validators have upgraded their clients, which resolved the Deposit Issue! Thanks to the community for acting fast and being responsive!

TLDR: We would like to notify all validators about misconfigured validator clients that were detected earlier this weekend, causing validator clients not to activate as expected. The network operation is not affected and is running smoothly. Genesis Validators are requested to upgrade their Prysm clients to at least version 4.0.6 and the LUKSO CLI version 0.8.0 following this guide. The migration will be postponed by at least a week, to ensure no interference due to upgrading the clients.

Over the last few days, Vlad Lykhonis (@vladlykhonis) from our community found an issue when depositing a validator key to Mainnet. Our internal team retested the process and discovered that this issue is present in the current Mainnet. We then formed a research group with members of the community and determined the issue’s cause and resolution.

We found that deposits were registered by some nodes, but never moved to a pending or activated state due to a missing flag in Prysm nodes - which make up the majority of the network, started through the LUKSO CLI.

The nodes in question also run Prysm version 4.0.0. This version is not recommended to be used anymore, but was the released version at the time of the LUKSO network start and the release of the LUKSO CLI.

What is affected?

The network is running smoothly and transaction processing is working fine. The only issue the current network has, is during the activation of new validators, which are crucial for the longevity and healthiness of the network. Therefore, it is important that Genesis Validators upgrade their nodes over the next days to enable proper validator deposits.

Requesting the Genesis Validators to upgrade their clients

The validator deposit process is not working as expected due to the overwhelming amount of GV running Prysm 4.0.0 and the flag:

--contract-deployment-block 0

missing for prysm clients.

In order to resolve the situation, we recommend every GV to upgrade their LUKSO CLI to version 0.8.0 and clients to Prysm version 4.0.6!

To make the process simpler we prepared a guide that will walk you through the process of upgrading.

This issue clearly shows the need for client diversity and we hope that future validators will consider using other execution and consensus clients such as Lighthouse and Erigon, to further diversify the network’s clients. We do not recommend switching from Prysm to lighthouse currently, due to recent lighthouse breaking upgrades that need to be tested first.

Postponing the Migration

Given the current situation, it would be risky to start the migration due to the following reasons:

1. As Genesis Validators will be upgrading their nodes, there could be epoch finalization delays that could interfere with the migration operation
2. If people migrate LYXe to LYX and use the deposit smart contract to deposit validators, those LYXe could be stuck, if Genesis Validators do not re-sync their nodes

For the above reasons, it is in everyone’s interest to postpone the migration by at least a week, until the current issue is resolved and tested.

We also took down the deposit launchpad website for the time being to prevent people from mistakenly depositing LYX to the deposit smart contract (though, this can be always done through the blockchain directly, as the deposit smart contract is permissionless and not pausable).

This issue shows us how we have to work together as a community to run and improve a decentralized network. As the LUKSO Blockchain GmbH and FNCE (Foundation for the New Creative Economies) are not running a single validator, we can only help as a coordinator in such moments to help the community resolve the issue.

We will monitor the network and do additional tests after Genesis Validators updated. An update on the migration and the resolution of the validator deposit issue will follow.

Let’s work together to make LUKSO the foundation for the next generation of blockchain users!