September 13, 2020: Idena validation session incident report

Description:

On September 13, 2020, during the validation session the Idena blockchain experienced a technical issue: The network stopped producing blocks and only empty blocks were mined. The blockchain nodes were out of sync. When trying to synchronize, connections between nodes were blocked and timed out due to the limit on the number of messages that could be transmitted simultaneously (10k messages). Each node, when trying to transmit messages, blocked reading messages from other peers, which led to a mutual deadlock between the nodes.

The Idena team released version 0.21.7 of the node which allowed an increased limit on the number of messages between peers and excluded flip keys messages while syncing. When about 1/4 of the nodes were updated, the network was able to produce blocks and mine the results of the validation.

Impact:

1. Mutual blocking of nodes

2. Delaying validation results

Root cause:

Node process is being blocked when pushing messages to peers. As a result, even nodes running on servers with a good channel were slowed down (and subsequently blocked) by slow nodes. As a result of the node lagging accumulation, most of the nodes were out of sync, the message queue accumulated and the entire network was deadlocked.

Possible solutions:

Non-blocking messaging to peers

A new release with the fixes is planned before the next validation session.