Idena hard fork announcement

We plan to release a new version of the Idena node on June 6, 2020. It will be a hard fork update. There are the following changes planned:

1. Preventing the issues that caused the last validation incident
2. Additional protocol improvements

1. Preventing the issues that caused the last validation incident

The problems identified during the last validation session showed that the network does not cope with large block mining.

Fig. 1. The last blocks of epoch #0045

During the long session, from 13:32:34 UTC to 13:38:34 UTC the network was not able to mine a single non-empty block. This happened due to the fact that a large block candidate could not be broadcasted over the network and reach all the committee members in time. The committee members, in the absence of a candidate block, voted for an empty block and waited for the next candidate block. The situation repeated with every next block until at the end of the long session a non-empty block was finally mined (14:02:13 UTC). The second large block was mined 2 minutes later.

Due to the fact that the second large block was mined after the end of the validation session, all the answers included in this block were not taken into account when calculating the validation results.

Measures planned to prevent the occurrence of this problem in the future:

1. Reducing the maximum block size to 300Kb
2. Reducing the number of candidate blocks proposed at the same time (from 10 to 5)
3. Distributing publication of the short session answers within 1 minute after the start of the long session
4. Reducing the size of the transactions with short session answers by removing some data and adding it to the transaction with long session answers

Measures to exclude validation failure even in case of the incident recurrence:

1. Validation session results will be calculated only when the last 5 non-empty blocks have been mined without ceremonial transactions. This will guarantee that all answers sent will be taken into account.
2. To switch the epoch, an empty block will have to be mined so that all the nodes have time to calculate the validation results.
3. Indexing transactions with the short session answers in mempool to display keywords on the long session without a need to wait for block mining (in case of an incident recurrence)

***

The reason for the incident could be that a large part of the network is running nodes on weak hardware (weaker than we expected) or running multiple node instances on a single weak VPS.

The Idena team apologizes for the incident that affected so many loyal Idena participants. We do our best to perform regular load testing on a large number of nodes and to allow the network functioning on hardware with characteristics available for the widest possible range of users. We estimate a feasible limit for the network growth is about 8–10k mining nodes. Further network growth will require sharding.

2. Additional protocol improvements to be included in the hard fork

1. RLP data serialization will be changed to Protobuf (2 times faster than RLP). RLP will be temporarily supported for backward compatibility. Instructions for migrating to Protobuf are to be published.
2. `Badgerds` profile will be used for new nodes (optimization for input /output operations with disk for IPFS).
3. Bug fix: Enormous fee for the flip deleting transaction (fee rate will be decreased 5 times).