Lition Network Status Update

As most of you are already aware, our Blockchain became stuck and stopped mining blocks at #242545 on the 24th of October. Our development team assessed the situation and implemented an appropriate hotfix that we believed would address the issue in a relatively quick manner.
A test network was then deployed to confirm that the network could remain stable after the hotfix. This test network consisted of 5 nodes that were deployed on web servers and the initial results were positive. After further stress testing however, the test network became progressively more unstable as more nodes were added. For this reason, we decided that it was in everyone’s best interest to delay the mainnet restart until this issue was fully resolved.
We’ve made significant progress since then and due to our latest development efforts we are able to announce that we are going to restart the blockchain in constrained settings by next week and our use cases will continue normal operations.
IBFT Consensus Problem
The bug that led to the initial stoppage of our Blockchain can be backtracked to a problem in our IBFT consensus mechanism. Nodes suffered under different transmission delays and this led to a situation in which nodes lock on different blocks. In such situation, it’s possible that the system doesn’t reach consensus of more than 2/3 nodes. If this is the case, the consensus process gets stuck and leads to a fail-stop of all nodes.
This issue was first reported by JPMorgan’s own Blockchain project, Quorum, and is a fundamental flaw in the IBFT consensus. Further details on this issue is outlined on a research paper from Roberto Saltini and David Hyland-Wood from Consensys (link). We are currently working on a fix for this issue but due to its degree of complexity we expect it to be several weeks until we can successfully ship the bug fix.
In the meantime, we are going to re-launch the network by next week with a limited number of nodes that can join the network. This number will be set at 11 for now. Furthermore, features will be implemented that ensure an immediate and automatic restart of the Blockchain at the latest produced Block in case the IBFT bug occurs again. We don’t anticipate the Blockchain having any issues again under consideration of the newly implemented limitations but we obviously want to be prepared for this scenario just in case.
Energy Use-case
The Lition energy use-case is currently being run parallel on a centralized version of our Blockchain similar to our testnet, in which latency time is equal to zero due to us running it on one machine. This way, we are preventing the IBFT bug from occurring as node transmission delays are non-existent. This Blockchain was launched as a precautionary measure in September and has been running in parallel since then in case of emergency. Our energy customers have therefore not been affected by the recent downtime of our decentralized network. This is crucial, as there are now thousands of active energy consumers and about 30 producers that trade daily.
Additional Use-cases
The IBFT consensus issue outlined above only occurs in sidechains with multiple distributed nodes that communicate a large number of transactions at high-latency. Business customers that do not need these requirements can therefore still use the Lition blockchain for their use cases. For now however, we recommend to only use the network with test data or to operate redundantly when real customer transactions are involved until the bug is fixed.
In Closing
Our developers are working around the clock to fix this issue and ensure that a flawless system is delivered. We appreciate all the patience and support that we have received from the community. Rest assure that we are 100% committed to making this phase of the project a complete success.

