Harmony Protocol Technical Quarterly Report and Roadmap for Q2 2023

Executive Summary

In Q1 2023, Harmony Protocol made significant progress in Stream Sync, State Sync, Cross-Shard Transactions, and Leader Rotation, along with numerous protocol updates and enhancements. As we move forward into Q2 2023, our roadmap focuses on State Pruning, Light Client development, and several new initiatives to optimize the Harmony network. However, it is essential to emphasize that these new incentives are recommendations and will not be implemented without community involvement and governance voting.

Key initiatives under consideration include downscaling from 4 shards to 1, reducing nodes from 250 to 100, reallocating annual token emissions with a 90% focus on ecosystem development, and improving transaction finality from 2 seconds to 0.8 seconds. Additionally, validator optimization and potential Boneh-Lynn-Shacham (BLS) signing key upgrades are planned to enhance efficiency, security, and cross-chain compatibility. These new incentives are designed to encourage new validators to enter the protocol, as they promote a more diverse and competitive environment that rewards innovation, efficiency, and dedication to the Harmony ecosystem. By attracting fresh talent and fostering a healthy ecosystem, Harmony aims to bring positive enhancements to the system.

Harmony Protocol’s primary goal is to bolster network performance, foster a thriving ecosystem, and drive further adoption within the blockchain space while ensuring that the community’s voice is heard and respected in the decision-making process.

Q1 Recap: Key Achievements and Initiatives

1. Stream Sync: We introduced an advanced synchronization process called Stream Sync, which builds upon the existing State Sync mechanism to improve network efficiency and performance further. Stream Sync leverages cutting-edge algorithms and data structures to minimize data exchange between nodes, reducing network overhead and accelerating new nodes’ integration. Using a pipelined approach, Stream Sync processes multiple state chunks concurrently, significantly reducing the time required for nodes to catch up with the current network state. This improvement ensures the availability of up-to-date and accurate data across the network, benefiting dApp developers and end-users. The development updates for Stream Sync include the implementation of versioned Merkle Trees for better data integrity verification, optimized caching mechanisms to expedite state data retrieval, and the introduction of advanced data compression techniques for efficient data transmission. The Harmony team has also focused on enhancing fault tolerance and reliability to ensure a seamless and robust synchronization experience.
2. State Sync: Harmony’s enhanced state synchronization, known as State Sync, is crucial for maintaining network performance and data consistency. It uses optimized algorithms and data structures, such as Merkle Trees, to minimize the data exchanged between nodes during the synchronization. The process involves dividing the state data into smaller chunks and synchronizing them in parallel, reducing the time required for nodes to catch up. Harmony accelerates new node onboarding and ensures data availability across the network by implementing efficient data compression techniques and prioritizing critical state data during synchronization. Development updates in State Sync include the introduction of Staged Stream Sync v1.0, which further improves performance and reliability, and implementation of caching mechanisms for archival nodes to reduce data access times.
3. Cross-Shard Transactions: Harmony’s development team has been working on multiple aspects of cross-shard transactions to improve performance and user experience. One key element is the implementation of composable commits, which allows transaction outputs from one shard to be used as inputs in another shard. This enables seamless interoperability between shards and maintains transaction atomicity. Harmony also employs efficient routing algorithms to minimize communication overhead between shards and reduce transaction latency. Shard coordination is improved through atomic cross-shard commit protocols and two-phase commit mechanisms, ensuring transaction consistency and finality across shards. Recent updates include planning a hard fork to implement chain ID fixes, cross-shard transaction optimizations, and imitating Ethereum’s syncing behavior in Harmony’s RPC calls for a more familiar developer experience.
4. Leader Rotation: The updated leader rotation mechanism in Harmony Protocol enhances network security and stability by ensuring that the next leader is responsive and available before the rotation process begins. This is achieved by introducing aliveness checks, verifying the next leader’s responsiveness and availability, and mitigating the risk of selecting an unresponsive or malicious leader. Harmony also employs a more secure and unbiased algorithm for leader selection, utilizing Verifiable Random Functions (VRFs) to generate unpredictable and unique leader candidates. This method strengthens the network’s resilience against attacks and collusion attempts. Development updates include the implementation of aliveness checks, improvements to the reshuffling process for shard leaders, and enhancements to the leader rotation mechanism by verifying the next leader’s availability before transitioning.

Protocol updates during Q1:

• Staged Stream Sync v1.0: Introduced a new version of the state synchronization mechanism to improve performance and reliability.
• Multiple SDK dependency updates: Various software development kit dependencies were updated to ensure compatibility and address security vulnerabilities.
• Feature registry: Implemented a system for registering and managing features, allowing better control over protocol enhancements.
• Enable caching on archival mode: Improved the performance of archival nodes by enabling caching, reducing the time required to access historical data.
• Fix epoch chain initialization issues: Resolved problems related to epoch chain initialization, ensuring smooth transitions between epochs.
• GoLang upgrades: Updated the Go programming language used within the protocol, benefiting from performance improvements and bug fixes.
• Fixes to config migration issues: Addressed issues with configuration migration, ensuring smooth updates and protocol changes.
• Add logging for NthNextHmy panic: Enhanced logging to help diagnose and address issues related to NthNextHmy panic.
• Resharding Leader updates: Improved the reshuffling process for shard leaders, increasing network security and stability.
• Schedule hard fork for chainid_fix and cx_one_transfer_precompile: Planned a hard fork to implement important protocol updates, including chain ID fixes and cross-shard transaction optimizations.
• Imitate eth_syncing behavior: Mirrored Ethereum’s syncing behavior in Harmony’s RPC calls, providing a more familiar experience for Ethereum developers.
• Fix self-query issues: Resolved issues related to self-querying, improving the reliability of node communication.
• Multiple Docker and GitHub action updates: Updated various Docker configurations and GitHub workflows to streamline development and deployment processes.
• Fix the max height issue: Addressed a problem calculating the maximum blockchain height, improving overall chain stability.
• Check the next leader’s aliveness: Enhanced the leader rotation mechanism by verifying the next leader’s availability before transitioning.
• Add TriesInMemory Flag: Introduced a flag to control the number of tries stored in memory, optimizing memory usage.
• Add Gas Price Oracle User Configuration: Enabled user configuration of the gas price oracle, allowing for more flexible and dynamic gas pricing.
• Fix p2p host discovery for legacy sync: Resolved issues with peer-to-peer host discovery in legacy sync mode, improving network connectivity.
• Fix state handling of self destruct: Corrected the handling of self-destruct operations, ensuring proper state updates and contract termination.
• Fixed Bitmap race error: Resolved a race condition in bitmap operations, enhancing network stability and performance.
• Removed unused methods: Eliminated deprecated and unused methods, streamlining the codebase and reducing potential security risks.
• Rotate external validators: Implemented rotation of external validators, promoting decentralization and network security.
• Add configurable HTTP and eth_call timeouts to RPC: Users can configure HTTP and eth_call timeouts, providing greater flexibility and performance tuning.
• Do not return extcode for validators: Improved privacy by preventing the exposure of external validator codes.
• Removed multiple engine dependencies: Simplified the dependency structure by removing unnecessary engine components, reducing potential conflicts and vulnerabilities.
• Fixes to context passing: Corrected issues with context passing, improving the overall reliability and efficiency of the protocol.

Q2 Objectives and Initiatives

1. State Pruning: Implement an efficient mechanism that utilizes data structures like Merkle Patricia Trees and reference counting to track and remove obsolete state data. This process will help reduce node storage requirements by eliminating unused or outdated information while maintaining consistency and data integrity. By periodically pruning the state tree, nodes benefit from lower storage and memory overhead, faster synchronization times, and improved overall performance.
2. Light Client Development: Continue the development of light clients by incorporating features such as succinct block headers, Merkle proofs, and Bloom filters, which enable lightweight clients to verify transactions and access blockchain data without downloading the entire chain. These clients, tailored for resource-constrained devices like smartphones and IoT devices, will improve performance by minimizing bandwidth and storage requirements. In addition, implementing protocols like the Light Ethereum Subprotocol (LES) and the Ultra-light Beam Synchronization (ULC) will ensure fast and efficient syncing of light clients, providing secure and accessible blockchain interaction for a broader range of users and devices.

New Proposed Initiatives:

• Downscale from 4 shards to 1 shard and 250 to 100 nodes.
• Restricting validators to use one BLS key per node.
• Split annual token emission of 441M as 10% to validators and 90% to the ecosystem.
• Improve transaction finality from 2 seconds to 0.8 seconds.
• Optional: Upgrade validator Boneh-Lynn-Shacham (BLS) signing keys, before the next Ethereum upgrade, for efficient bridges and light clients.

As we continue to refine and optimize the Harmony Protocol, we believe involving our dedicated community in critical decision-making is essential. To ensure that the interests of our community members are taken into account, we will provide an opportunity for you to vote and give feedback on the proposed tokenomic changes, including the 90% and 10% reward adjustments, as well as the reduction in the total number of nodes to less than 100. Your input and participation in these crucial decisions will help us shape the future of the Harmony Protocol, aligning it with the collective vision of our growing community.

The proposed tokenomics and node reduction changes are designed to foster a more sustainable and dedicated validator ecosystem within Harmony. By prioritizing long-term, committed validators, we aim to create an environment that encourages homebrew validators to run on bare metal infrastructure instead of relying on cloud server usage. These changes are intended to motivate and empower validators who actively contribute to the Harmony ecosystem by developing dApps, developer tools, exchanges, and other valuable use cases.

We recognize that these adjustments might discourage some current validators; however, our primary focus is empowering those who genuinely believe in Harmony’s long-term vision and success. By supporting and promoting validators that bring value to the network and invest in its future, we can create a stronger, more vibrant community that propels Harmony towards its ultimate goal of widespread adoption and innovation.

Downscaling and Token Emission

In Q2, one of our primary initiatives will be to restructure the Harmony network by consolidating from 4 shards to a single shard and simultaneously reducing the number of nodes from 250 to 100. This strategic move will have several consequences for the network regarding technical performance and operational costs.

Reducing the number of shards and nodes will lead to a 2.5x decrease in the total elected validators in the network, streamlining the consensus process and reducing the communication overhead. This consolidation will also lower the validator operations budget, as fewer nodes will need to be maintained and secured, allowing for a more efficient allocation of resources.

Reducing the number of nodes from 250 to 100 will improve message propagation and consensus speed, potentially enhancing the network’s transaction throughput and finality time. This is because fewer nodes will participate in the consensus process, reducing communication complexity and speeding up the overall process.

While a reduction in nodes may raise concerns about the potential impact on decentralization, it is essential to note that the Harmony Protocol employs a secure and efficient sharding mechanism that mitigates the risk of centralization. The network can maintain its decentralization and security even with fewer nodes by effectively distributing the transaction load and validator responsibilities across the single shard.

In conclusion, downsizing the number of nodes in Harmony Protocol can lead to performance improvements without inherently compromising decentralization, provided that the proper mechanisms are in place to maintain a secure and distributed network. This can be achieved through community engagement and transparent governance mechanisms that encourage a healthy balance of power within the network. Ensuring a broad and diverse distribution of delegations among the remaining validators will further support decentralization.

Regarding the annual token emission of 441M, we plan to allocate 10% of this amount (44.1M tokens) to validators and the remaining 90% (396.9M tokens) to ecosystem development. This distribution will ensure that validators remain fairly rewarded for their contributions to the network’s security and stability, even though their overall rewards will be reduced. The focus on ecosystem development will accelerate depegged asset recovery and drive developer growth, ultimately attracting more projects and users to the platform.

In the future scenario, with the reduced 10% allocation to validators, the average earnings per slot for validators will still be maintained:

44,100,000 tokens / 100 nodes * $0.02 per token = $8,820 average earnings per slot.

By readjusting the token distribution, we can maintain a healthy balance between incentivizing validators to maintain high-performance nodes and providing ample resources for the growth of the Harmony ecosystem. This approach aims to achieve sustainable network growth and adoption while keeping the network secure and decentralized.

The Q2 initiative to downscale the network will increase operational costs and technical performance efficiency. By carefully allocating token emissions, we aim to strike a balance between incentivizing validators and fostering a thriving ecosystem, ultimately contributing to the long-term success of the Harmony Protocol.

Transaction Finality Improvement

We aim to improve transaction finality from 2 seconds to 0.8 seconds by optimizing the consensus mechanism and network communication. Transaction finality speed is influenced by various factors, including the number of nodes participating in the consensus, the volume of network messages exchanged, and the shard leader’s efficiency in computing aggregate signatures.

In the existing setup with 250 nodes, network latency, and communication overhead contribute significantly to the 2-second finality time. However, it is important to note that with the current load, shards 1, 2, and 3 can achieve 1-second finality, but the system is intentionally set to wait for 2 seconds. By reducing the number of nodes to 100, we expect to decrease the finality time to 0.8 seconds, as calculated by the formula: 2 / (250/100) = 0.8 seconds. This reduction in nodes decreases the number of required message exchanges and limits the communication overhead, leading to a faster consensus agreement. The modified setup would allow us to capitalize on the existing performance capabilities of the individual shards, further optimizing the overall network performance and transaction processing times.

In addition to the node reduction, we will explore other optimizations to enhance the shard leader’s computation of aggregate signatures. These optimizations can further reduce the time required for the shard leader to compute and validate signatures, thereby accelerating transaction finality. For instance, we can investigate using more efficient cryptographic algorithms, parallel signature verification, and batching techniques.

We also plan to analyze and improve the underlying network infrastructure, such as optimizing peer-to-peer connections, implementing more efficient message dissemination protocols, and utilizing network compression techniques. These improvements will help reduce network latency and further enhance transaction finality.

In summary, by decreasing the number of nodes, optimizing shard leader computations, and enhancing the network infrastructure, we aim to significantly improve transaction finality from 2 seconds to 0.8 seconds, offering a more responsive and efficient blockchain experience for Harmony Protocol users.

Validator Optimization and BLS Key Upgrade

In Q2, we plan to re-align validators by adjusting the distribution of the foundation treasury’s tokens, which includes the current 305M tokens ($6.2M) staked from the foundation. The new delegation will allocate 1.20B tokens ($24M) among the validators. This optimization will consider validator performance, security measures, and uptime to ensure a fair and effective distribution. By rebalancing the delegated tokens, we aim to maintain a robust and decentralized network with evenly distributed voting power, reducing the likelihood of cooperation or centralization.

We are also exploring options for using the rewards generated from staking these tokens for initiatives such as depegged asset recovery, supporting DApps, and fostering developer growth within the Harmony ecosystem. By directing some rewards towards these areas, we intend to create a vibrant and sustainable environment for developers and users, driving further adoption and innovation in the blockchain space.

In addition to validator optimization, we are considering upgrading the validator Boneh-Lynn-Shacham (BLS) signing keys in addition to validator optimization before the next Ethereum upgrade. Upgrading to BLS signatures offers multiple benefits, primarily due to their unique aggregation property, which allows multiple signatures to be combined into a single, compact signature. This reduces the size of the blockchain’s signature data and decreases the computation required for verification, thereby enhancing the performance of cross-chain bridges and light clients.

Harmony Protocol can achieve more efficient communication between Ethereum and other blockchain ecosystems by adopting BLS signatures, resulting in faster and more secure cross-chain transactions. BLS signatures have a deterministic and constant-time verification process, which reduces the risk of timing attacks and improves the protocol’s security. This upgrade will ensure that Harmony stays compatible with future Ethereum upgrades, enhancing cross-chain compatibility and user experience.

The upgrade process will require thorough testing and benchmarking to ensure compatibility and seamless integration with the existing protocol. Moreover, this optional initiative will demand coordination with the validator community to facilitate the smooth transition to the upgraded BLS signing keys.

By re-aligning validators and potentially upgrading to BLS signing keys, Harmony Protocol aims to bolster network efficiency, security, and cross-chain compatibility, ultimately providing an enhanced user experience and fostering greater adoption in the blockchain space.

Conclusion

In Q1, Harmony Protocol made significant progress in State Sync, Cross-Shard Transactions, and Leader Rotation, alongside numerous protocol updates and improvements. As we move into Q2, our focus shifts to State Pruning, continued Light Client development, and new initiatives such as downscaling the network, improving transaction finality, and re-aligning validators. We aim to strengthen the Harmony Protocol ecosystem by achieving these objectives and driving further growth and adoption in the blockchain space.

Photo by Thanos Pal on Unsplash

Pros of Moving from 4 Shards to 1 Shard:

• Simplified consensus process: With a single shard, the consensus process becomes more straightforward to manage, improving overall efficiency.
• Reduced communication overhead: Less inter-shard communication will be required, minimizing network overhead and enhancing performance.
• Lower validator operational costs: Validators will only need to maintain nodes for a single shard, reducing operational and maintenance costs.
• Easier developer experience: Developers will have a more accessible time building and deploying dApps on a single shard without worrying about cross-shard transactions.

Cons of Moving from 4 Shards to 1 Shard:

• Potentially reduced scalability: One of the critical features of sharding is increased scalability. Moving from 4 shards to 1 may raise concerns about the protocol’s ability to scale efficiently in the future.
• Negative community perception: The community might perceive this change as a step back from the original vision of Harmony’s scalable infrastructure. They might question the decision to scale down, particularly after the significant work that has gone into developing cross-shard transactions.
• Reconfiguration of existing dApps: dApps built to work with multiple shards may require updates and reconfiguration to function correctly on a single shard.
• Possible centralization concerns: Reducing the number of shards may raise concerns about increased centralization, as fewer shards can lead to a higher concentration of validators in the single shard.