Latest Advances on NEO — Operational Perspectives
While researchers have been extensively investigating novel consensus paradigms, the real implementation of those mechanisms is limited when real-world systems are the focus. Pseudo-codes are usually the main guidelines for turning that into reality, thus, an arduous and meticulous task is left to programmers.
Following the same reasoning, one can propose several smart and complex negotiation protocols for multi-agent systems. On the other hand, when we look at reality, vehicles (massive use of IoT devices) are not yet smartly communicating with traffic lights and sensors during their routes, and there is still a gap between the academic literature and the construction of these protocols in our digital/smart cities.
In this brief note, we are going to highlight the brilliant advances on Neo core library, with better timings and deep attention on the details of a simple Byzantine Fault Tolerance implementation.
A journey inside Neo dBFT Consensus protocol
While other developers can be in favor of other complex, and claimed, more efficient consensus mechanism, NeoResearch team has been defending the fine-tuning of the current dBFT. Inspired by the well-known literature mechanism Practical Byzantine Fault Tolerance — pBFT, a simple protocol proposed around 1999 by Castro & Liskov, the current Neo dBFT improves that concepts to the Blockchain reality.
In particular, two groundbreaking changes are going to the highlighted here: one related to an impressive achievement of Neo Consensus stability and a second related to a rare situation that is currently being solved.
1 — Efficiency using the Akka framework — Neo 2.9.0
Mostly, developers have in their hands pseudo-codes and previous experience (for themselves or from the beautiful sea of open knowledge that, nowadays, we have).
The abstraction for each specific case is done by the imagination of those who conceptualize that. In the old time, while programming C/C++ Optimization codes, memory leaks were hard to deal when dealing with pointers and references. On the other hand, the great advance of tools such as Valgrind made the testing and verification tests easier.
On the same line of reasoning, parallel computing techniques have been evolving. The ability to use them and visualize where each technique or framework should be applied is the magic. That is why we are thankful to Erik Zhang for his brilliant dedication to the project and great catch that he has been having in improving Neo in a precise manner.
Improving the consensus response times
While understanding the stages of the Consensus procedure can be easy to some experts, fully comprehend the effort in each step is not a trivial aspect.
That is why Igor focused on developing a simple tool for drawing consensus messages (states), https://github.com/NeoResearch/neo-tests/tree/master/consensus-draw.
In Figure 1, we can see that the each step of the Block Generation (phases of the dBFT) were taking some time to reach other nodes, even in a local environment.
After seeing and analyzing the graphs, such as the one present at Figure 2, we even needed to change the CN Logger to milliseconds, on 13th of August, because precision has improved around 100x to 1000x!
Why did it happen?
Some important aspects are:
- Better control of the communication between nodes, using: smart ordering and filters.
We mention smart ordering in terms of queuing the messages regarding priorities. Differently than other protocols, reaching a consensus requires several types of communication and a “smart conversation” between agents. We can see this family of protocol as an environment composed of several selfish autonomous agents.
At this point, one can imagine that each agent can prioritize its own objectives. Furthermore, each agent can perform a specific task on the network.
It is noteworthy that the current improvements were able to better organize these broadcasted messages throughout the network. Thus, each participant is now much more able to get exactly what is needed to accomplish a global objective, which is: generate the blocks in a trusted manner.
It is a optimal solution? No, we do not think so, there is a huge sea of possibilities for optimization and, perhaps, better communication fine-tuning. In this sense, we are still motivated to keep understanding and studying new strategies for making this “simple” consensus to work as a state-of-the-art example of how agents can communicate in an efficient massively decentralized P2P network ecosystem, the Neo Blockchain.
2 — “fork” tests
The first implementation of the dBFT did not considered, for the sake of simplicity, a commit phase of those validators involved in the block generation.
Shargon, Belane and Erik already worked on the prototype of an additional phase aligned with the reasoning of the dBFT: https://github.com/neo-project/neo/pull/320, which is now being ported to meet the currently changes made to Neo 2.9.0 clients.
In a extremely rare situation, with delay and several communication losses between nodes, a specific node could stay with a “forked” block. The main reason is:
1 — because of the nonce that is generated differently by each primary;
2 — and/or the possible new coming transactions that arrived while that specific node lost connection (exactly when the necessary amount signatures was achieved and the node assumed that all others would follow the same block).
An effort done by the Security team and NeoResearch was to reproduce such scenario (a difficult task, mainly because of the fine-tuned communication with the Akka refactoring). For this purpose, the scripts available at https://github.com/NeoResearch/neo-tests/tree/master/forking-tests were created.
On the night of 21th, August, NeoResearch team was able to concisely reproduce such scenario (only interfering in external conditions such as network delays), opening doors for facilitating the validation of the ongoing solution, e.g, automated testing.
3 — Fine-tuning the consensus time
We imagine that everybody in the community expect the blocks to be carved always within 15s (one can surely expect a big improvement after 2.9.0 entering the mainnet, however, there is still some possibilities for ensuring a precise block time in most of the time). That is why a precise fine-tuning is currently going on at https://github.com/neo-project/neo/pull/268.
We believe that with some precise changes and a kind of Kalman Filter we might be able to ponder trade-offs that are inherent to the system.
It is a great pleasure to share some insights with all of you here.
Finally, we would to really thanks this brilliant and united community that has been essential for ensuring improvements aligned with the desires and wishes of several decentralized involved agents.
