Open Access to Harmony
Introduction
Harmony strives to create a decentralized platform fully externalized and self-sustaining. Harmony will offer the required infrastructure and internal services until the desired state of the network is achieved. This article aims to provide open access to Harmony’s cost analysis, projections, and security practices.
Cost Overview
Below is a screenshot of Harmony’s monthly costs from November 2022 to the projected cost of June 2023. As captured below, Harmony has been working to save costs while ensuring the resilience of the network. We were able to reduce the total cost from $120,000 to $51,000, and the current projected cost for June 2023 is approximately $15,000 (note that these projections can change as more or fewer demands are required). Migration from AWS and termination of underutilized instances and plans have been and will be the major contributing factors to the decrease in cost.
The initiative of these activities is not to simply reduce costs but rather to pave a path for a self-sustaining, decentralized network. This has been Harmony’s driving mission from the start. In order to achieve the goal, adjustments to our services in accordance with the current need of the network are a must.
Migration from AWS
Before the major migrations and terminations, the majority of Harmony’s services were concentrated in our two AWS accounts: “development” and “main”. The purpose of the development account is for engineers to develop and host services before landing in the main account. The purpose of the main account is to host services that were fully developed and fully integrated into the network. Because new features and services were continuously developed and tested, the development account hosted a lot more services than the main one, explaining the difference in cost between the two accounts.
While AWS offers many conveniences, it is more expensive than other providers. The main reason is the data transfer cost AWS charges. The majority of other providers do not charge data transfer costs, thus offering more affordable options. AWS does offer various savings plans to mitigate data transfer costs. However, with the decline of traffic in the Harmony network, we are not able to fully utilize the plan and make long-term commitments.
The reasonable choice was to migrate our services from AWS to other providers. Hetzner, Digital Ocean, Latitude, and Contabo are the alternatives Harmony has chosen. These offer a more appropriate pricing model at the current state of Harmony, allowing us to be prudent with our resources. This allows us to diversify and adopt a multi-cloud strategy. Harmony plans to fully migrate out of AWS and utilize the aforementioned providers.
Multi-Cloud Strategy
The main reason for hosting our services in multiple cloud providers is to prevent a single point of failure. Though the chosen platforms are reliable providers, the chance of failure is never 0%. To prevent services from going down all at once and causing a network outage, Harmony’s services have been distributed to various data centers in different providers.
The breakdown of Mainnet services is shown below:
As shown above, Harmony still hosts a considerable amount of services in AWS despite the effort to migrate away from AWS. This is because Harmony purchased a yearly subscription to Compute Savings Plan and Reserved Instances in 2022 with the cost of approximately $12,000 per subscription per month. Last year, when Harmony was handling an immense amount of traffic, purchasing the subscriptions was a reasonable choice. The plans were utilized to the fullest extent and Harmony was able to reduce expenses. However, with the decreased usage and traffic, Harmony cannot fully utilize the subscriptions.
The Compute Savings Plan and Reserved Instances end in April and June of 2023 respectively. Because Harmony is fully committed to the savings plan until April, there is a limitation on the actions Harmony can take. Thus, services affected by it will be the latter priorities in migration. On the other hand, Harmony is putting significant effort into handling the Reserved Instances.
Termination of Underutilized Services
Unlike the Savings Plan which targets the majority of the instances, the Reserved Instances were mainly for elastic RPC clusters. As Harmony was handling a significant volume of requests in the previous years, there was a demand for elastic RPC clusters. Twenty i3en.xlarge were purchased as Reserved Instances and the elastic clusters were established to improve the performance and scalability of the network, despite their greater cost. The clusters were fully utilized and Harmony was able to serve the network requests in a more efficient manner.
However, with the decline of traffic, elastic RPC clusters are no longer needed. The clusters were not fully utilized thus maintaining the services would be simply wasting resources. Harmony has retired elastic RPC clusters and fully transitioned to the legacy RPC nodes (at the time of publishment, Harmony is supporting three full RPC nodes). Along with the clusters, more than a hundred shard 0 explorer RPC nodes have also been retired as they were underutilized as well. Mind that Harmony will always transition and scale to fulfill the network needs. However, with the current situation, the legacy RPC nodes can perfectly handle requests of the network.
The elastic RPC clusters occupied the majority (17 of 20) of the Reserved Instances. Now that the clusters have been terminated, Harmony has listed the Reserved Instances for sale. This would reduce the economic burden of resources that are not being utilized. One might wonder whether the Reserved Instances can be occupied by other services. However, currently, there are no services in Harmony that require as much redundancy and computing power. Thus, the only reasonable option is to terminate the services and list the Reserved Instances for sale. Additionally, listed instances will still incur data transfer costs.
Along with the elastic RPC clusters, Harmony terminated other underutilized instances and cut redundancies. One major redundancy we have reduced is the archival nodes. Originally Harmony supported four archival nodes, but Harmony has decided to maintain only three nodes. An archival node from Hetzner has been terminated because the four original archival nodes were minimally utilized. Currently, three nodes are enough to handle the functionality while preventing a single point of failure. With other required services, we have exercised compute instance resizing activities to suit the network need.
Harmony is consistently monitoring and inspecting the health of the Harmony network. We are striving to ensure the stability and robustness of the network. Harmony will work full hands to scale and provide the necessary services whenever more demands are required.
Security Practices and Measures
Harmony has been striving to improve and reinforce the security of our cloud infrastructures. Infrastructure changes, mainly enforcing stronger imposition of Single Sign-On (SSO), Multi-Factor Authentication (MFA), and Virtual Private Networks (VPN) have been executed. SSO allows users to sign in to multiple applications with a single set of credentials, preventing multiple credentials from being created. MFA adds an extra layer of security by requiring users to provide additional authentication factors, for example, codes sent to their personal mobile devices. VPNs provide secure access to our cloud resources by creating an encrypted tunnel between the network and the cloud environment. All these methods provide extra security layers to the ones already supplied by the cloud providers.
On top of the multiple security layers, we have been actively securing our passwords and secret keys. We have implemented an even more aggressive password and secret rotation, and API clean-up process. This helps prevent unauthorized access to our cloud resources. Passwords and secrets will be outdated quickly due to the active process, making it even more difficult for attackers to compromise the resources.
While the security layers and practices have been present in Harmony, we have reinforced the measures with even greater emphasis. The implementation of countermeasures at the protocol level and the adoption of these security strategies will further enhance the robustness and security of the Harmony network.
Harmony’s Roadmap
In order to provide a more secure, decentralized, and self-sustaining network, Harmony will be working on several key initiatives. First and foremost, we plan to complete our migration from AWS as previously mentioned. We will be able to reduce costs and enforce our multi-cloud strategy. This will help us avoid vendor lock-in and be prudent with our resources, as well as to prevent a single point of failure.
As we are migrating away from AWS, we hope to segregate the nodes and infrastructure in an orderly, secure manner. Currently, there exists only a simple distinction between the different services. This distinction is enough for us to organize our services. However, we hope to improve this organization method and create a clear distinction between the nodes and infrastructure of different projects. This would help our multi-cloud infrastructure become even more organized and secure.
Another key initiative is to create a centralized cost service for the costs of multiple cloud providers. One downside of a multi-cloud strategy is the difference in the pricing models of each provider. Due to the variation of the invoices, mainly the way items are categorized and priced, forming a single source of information in a uniform manner may be inefficient and confusing. Harmony plans to create a centralized cost service that will automate this process in an efficient and clear manner. This will not only enable both the internal and the community members to clearly visualize Harmony’s expenses. Transparency is a key mission Harmony believes in. We hope that the service will bring transparency to our community.
Last but not least, Harmony anticipates various protocol enhancements and the establishment of partnerships and collaborations that will allow the network to fully externalize. As stated, Harmony’s ultimate goal is to create a completely decentralized and self-sustaining network. These initiatives will bring us a step closer to those goals.
Conclusion
Open access is a key component in creating a decentralized community. Harmony hopes to bring more transparency to the internal works, projections, and costs to the community in order to collaborate for a self-sustaining network. Kindly stay tuned for further updates!
