Harmony Network Status Update: June 14, 2021

Harmony Community,

As many of you know, the growth of Harmony recently has challenged the network in many ways. We acknowledge that we failed in two critical points:

1. Our infrastructure wasn’t resilient enough to handle the increase in transaction volume.
2. We didn’t communicate early enough or as often as we needed to.

We know we could’ve done a better job at communicating with the community about the issues, what we’re doing about it, and an ETA on when things will be fixed. This post will start to do that.

What are the specific issues and how it affects users:

There are two main issues that happened during the last two weeks. First, the hugely increased transaction volume that started 2 weeks ago overloaded our RPC endpoint, making it unstable and degraded for production usage. Specifically, users reported failed transactions or unusable applications due to broken websocket or stale data from the RPC endpoint. The increased transaction volume also made it slower for transactions with low gas prices to be accepted.

Second, there was an incident of a P2P network flooding attack which significantly slowed down block time and temporarily interrupted block production for 2 hours triggered by a bug in double slash detection. Coupled with the RPC endpoint issues, the usability of the network was degraded further.

By the time of this writing, the second issue of network flooding is mostly resolved and the block time is now back to normal at around 2.0 to 2.1 seconds. We are still working on a few changes to further strengthen the P2P network and reduce transaction failure rate.

For the RPC endpoint issues, we’ve already added the rate limiter to RPC APIs so the backend won’t go out of memory. More work is in progress on the code logic level and ops level to make the RPC fully production ready.

What we’re are working on now:

Protocol level:

• P2P spam protection where rate limiters are added to prevent the rebroadcast of invalid messages.
• Improved transaction gas price model with a reasonable minimum gas price requirement and a hard-cap of 80 million gas limit on every block.
• Optimization on crosslink and staking reward distribution logic to reduce computation burden on the beacon chain.

RPC endpoints:

• Upgrade and add more production-grade end-point machines to handle the increased load.
• Explorer node backend efficiency fixes on transaction history API.
• Add monitoring on specific RPC API functionalities to make sure they are working properly.

ETA on when things will be fixed:

• We are actively working on this issue. You can track it on our Github: Explorer nodes can’t catch up with latest blocks.
• We will communicate updates on our progress on our official Twitter account. We are also working on a public status page for real-time network monitoring.

We have a responsibility to our application partners, validator partners, users, and our community. We aim to keep up our end of the bargain and do better. We know developers and partners have many choices on which to build their applications. Thank you for choosing Harmony and we are working hard to be maintain a reliable platform for validators, developers, and to the community.