Block Synchronization Mechanism on Neo
1. Introduction to block synchronization mechanism of Neo2
In Neo2, the process of synchronize block is more complicated. There are two steps: synchronize block headers and synchronize blocks.
As shown in the following figure, is the flow chart of synchronization mechanism of Neo2. Synchronize block headers corresponds to 1,2 steps:
1. When the local node finds that the local blockchain height is lower than the height of the remote node, it will send a GetHeanders (header hash) message, where the header hash is the hash corresponding to the block head with the highest height of local node.
2. After the remote node receives the request, it will return the specified number of Headers from the corresponding header in the header hash to the local node (the default maximum number of Headers requests is 2000).
After the block headers are synchronized, the blocks will be synchronized. This process corresponds to the steps in the figure 3–7:
3. After the block headers are synchronized, the hash corresponding to the local block with the highest height will be found.
4. Send the GetBlock(hash) message to the remote node based on the hash found.
5. When the remote node receives the request, it gets the hash corresponding to the starting block and the number of blocks requested (the default maximum number is 500). Then, starting from this hash, the corresponding hash values of the specified number of blocks are obtained in turn, and all the hashes are put into one hash[]. When the fetch is complete, the hash[] is returned to the local node as an Inv message.
6. When the local node receives the Inv message, it filters the hash[] in the Inv message and puts the filtered hash[] into the Getdata request and sends it to the remote node.
7. The remote node receives a GetData(hash[]) request, gets the corresponding block for each hash in the hash[], and sends the blocks to the local node.
2. Problems in block synchronization mechanism of Neo2
It can be seen that the block synchronization mechanism of Neo2 is relatively complex, with the following problems:
1. The pattern of synchronizing the block headers first and then the blocks has significant advantages in blockchains that may fork, because the headers synchronization is very fast and can quickly select longer chain. But the consensus algorithm used by NEO determines that its blockchain will not be forked. So the advantages of header-first mode have become obsolete. If we no longer use this pattern, we can save the network overhead of block headers synchronization.
2. The current GetBlocks command is inefficient and the name is misleading. By name, we will receive the blocks, but in fact, we will receive an Inv message containing a hash array. The hash array in the Inv message is then filtered, and the corresponding blocks are obtained using the GetData(hash[]) command.
3. The new block synchronization mechanism of Neo3
According to some problems in block synchronization mechanism of Neo2, we improved the block synchronization mechanism in Neo3 as follows:
1. The block headers synchronization mechanism was removed and the block height was used to synchronize the blocks directly instead.
2. Add GetBlockData(IndexStart,count) message, you can directly get the specified blocks from IndexStart, no need to get through Inv message, reduce the network communication overhead.
3. Added SyncManager to manage block synchronization tasks.
As shown in the following figure, is the flow chart of the block synchronization mechanism of Neo3.
It is mainly divided into the following steps:
1. First, the blockchain height is synchronized with the remote node via the PingPong message.
2. If the local blockchain height is lower than the remote node height, a GetBlockData message is sent to get the missing blocks.
3. The remote node sends the request blocks in turn to the local node.
4. SyncManger records the received block indexs.
5. Send the received block to the local node.
6. The local node validates the validity of the block. If it is a valid block, a Persisted Block Index is sent to the SyncManager, and the block synchronization process ends.
7. If it is an invalid block, send an Invalid Block Index to SyncManager, and SyncManager will reselect other nodes to assign the task. And mark the node that sent the invalid block.
4. Test and conclusion of new synchronization mechanism of Neo3
For the synchronization mechanism of Neo2 and Neo3, we tested the block synchronization speed under different conditions as follows:
P2P Sync time(seconds) with StatesDumper intalled (start from index 0)
Improve about 7 times.
P2P Sync time(seconds) without StatesDumper intalled (start from index 30)
Improve about 4~5times.
Conclusion: The new block synchronization mechanism effectively reduces the network communication overhead and greatly improves the speed of P2P block synchronization. It also lays the foundation for node health.
Note: The code related to the above scheme is still under review and does not necessarily represent the final design scheme.
By Xiaoyun Yang
