How KysenPool Dragon 🐉 surfaced with Harmony $ONE
How Decisions of KysenPool’s Validator setup helped strengthened our position to becoming a highly staked operator
A storm was brewing and it was the launch of the Harmony Open Staking Network. Unlike the successful MainNet launch for Cosmos, the stablecoin oracle pricings in Terra, the masternodes for Fusion, the slots in Cardano, and the trusted UNL in XRP. Harmony’s Open Staking felt a little … different.
KysenPool has been participating as a Foundational Node Operator of Harmony nodes since Day One to help Harmony operate nodes as a third party as a decentralization trial run to mature the consensus protocol. The trial run during the year helped Harmony’s team with stabilizing its protocol, engaging with a decentralized set of participants, while node operators get to practice running nodes, understanding the stress it puts on the nodes, build out tooling, automation, etc. — all to prepare to journey into a staking economy
Harmony’s Staking design included the concept of Effective Proof-of-Stake.
What the heck is that? Validators that get to the middle wins? This is a layer of complexity which Validators (a.k.a. Node Operators) need to wrap their head around… and fast.
The Trailer
The final stages of OpenStaking was coming to a close late in April 2020. The Stake Heist initiative was launched to incentivize Validators to attempt to stake and run validators in an adversarial environment.
Being a Foundational Node Operator, we participated a testnet Pangaea for a short while and then in the Open Staking TestNet during its first few weeks. There was a lot of churn and the network was reset a number of times. We figured out something but not everything was clear.
But we were able to practice the open staking procedures and had a sneak preview into what’s about to come.
“With practice comes mastery.
With mastery comes knowledge.
With knowledge comes strength.”
Sneak Preview
Then came May 12th. Harmony’s Open Staking was about to launch. Announcements were made. People started flooding in. No one can do anything just yet.
Then we cross into the first “day” of open staking. The start of an “epoch”. Validators, both newbies and experienced operators saw a combination of terminologies that usually don’t co-exist with one another:
- What happens during an epoch, especially the first one?
- What is a Shard and how do they relate to a BLS key?
- What about Slots and how do they relate to BLS keys or a Shard?
- How do BLS keys assign to a Shard and the go into a Slot?
- What is an Expected Return got to do with Uptime?
Oh my, the complexity!
Validators started popping into Telegram and Discrod. Early adopters started creating their Validators early to gather stakes really quickly. They all started promoting their strengths and watching their delegations grow. Most were helping, some weren’t as available. There was a mad dash to the finish line.
We had to scour through the chatter, documentation and blog posts to gather a lot of information, and discard disinformation. There was some level of confusion but not nearly as much as I’ve seen other chains go through. The documentations were set out to be clean but you had to watch for certain trip wires of invalid links and ill-defined parameters.
💥Boom… it’s Launched! 🚀
The network begins, we had everything provisioned and ready to go but had to clear the last hurdle 20 minutes into the epoch to begin signing with our Validator. Our uptime suffered initially but we recovered to over 90% “Uptime” within the same epoch thanks to our 99.99% blockchain signing rates reaching consensus really quickly.
We were one of the Validators that was created at a later block time before the start of the epoch because we spent a LOT of time making sure that make the best decision as a Validator. The best was to verify our hypothesis with how other Validators are defining their nodes. This was a ONE shot that we can’t afford to be wrong.
This is what we had figured out:
- Figure out how to generate a bunch of BLS keys with randomly generated passwords (see why this is important under Security below)
- Identify a set of generated BLS keys separating the private keys in a cold storage
- Organize them into the Shards that they belong in (0, 1, 2 or 3)
- Without containers, there’s a 1-shard per server rule — but each server can host multiple BLS keys of the same shard (multi-shard capability coming soon with a bug fix)
- Create the Validator with an initial BLS key
- Connect with known investors to delegate the initial stake
- Decide which shard to add into our Validator (Shard 0 = Beacon Chain)
- Edit the validator to assign n-number of BLS keys in x / y / z shards
- Architect how to securely deploy these keys into our Validator repeatably and reliably, with the ability to rotate new keys in
- Create Cloud initialization scripts to deploy quickly and effectively
- Hook them into health monitors automagically
- Create a fleet of servers which automatically restores degraded servers within minutes
- Tie in the auto-rebuild scripts with re-sync steps cloning from Harmony’s repository to achieve consensus again
- Run at least two instances of the same immutable setup to achieve High Availability (signing blocks at rate of 99.99% per epoch)
- Do this all from “Zero-to-BINGO” (signing blocks) in under 15 minutes
- Do all of the above for a really affordable rate
- … and oh yes, figure out what to name our Validator!
Decisions
This is what we decided before we created our Validator and soon made changes to after we operated the node for several epochs:
[BRAND] We created ONE validator, instead of multiple validators
We think having multiple validators identities will dilute the identity of our Validator, our purpose. Having multiple validators also puts us at risk of not having enough delegations to stay competitive. If we needed to stay within the hotly contested Effective Stake “zone” to earn high return rates, we can split our delegations via the multi-BLS capability in Harmony’s protocol.
Also, a single validator can be scaled horizontally by assigning the multiple BLS keys into multiple Shards and our underlying servers have the flexibility to reflect that. We didn’t know how well it will perform on a fairly small CPU footprint on a multi-BLS Open Staking environment but we took the learnings from our Foundational Node days, and a small bet at this after seeing initial results being quite promising.
[FLEXIBILITY] We built in abilities to move around to ANY shards
Automated deployment is key here. We built and rebuilt our automation scripts to be able to achieve this with a high degree of confidence. We practiced these in live Cloud-native environments, with multiple Cloud providers. We found a secure solution to safely store and retrieve BLS keys.
When a new epoch crosses over, we have the ability to switch into new Shards quickly and easily to avoid less profitable Shards. We do so by spinning up servers just in time, sometime minutes before the new epoch rolls over.
Our decisions aren’t always based on profitability. We also wish to help decentralize the Harmony Shards if it is in need of help (e.g. Shard 0 — the Beacon Chain — was “difficult to earn” for a while). This comes at a cost of Uptime and profitability. But we thought it was the right thing to do as long as it doesn’t hurt our expected yield too greatly.
[SECURITY] We decided to add a secure passphrase to our BLS Keys
We used a hashing algorithm alongside some unique key generation methods to generate hundreds of passphrases and added them into ALL of our BLS keys generation procedures.
This seems to be a given. However, BLS Keys can be created without a passphrase. Yes, that’s right. You can skip this step. You can place it into your node and start signing with non-passphrase secured key. Nothing wrong with it?
Wrong. If a malicious perpetrator decides to scan through every Validator’s BLS keys (it’s a publicly available datapoint) and then runs a modified version of Harmony to sign invalid blocks, a Validator could eventually be penalized when reported upon via slashing or even a full out ban.
[RESILIENCE] We decided to always run DUAL presence nodes
We will always spin up at least two nodes in different geographical regions to
- help propagate the spirit of a decentralized network
- improve our Uptime, our ability to sign new blocks on time
There are no penalties for signing the same blocks with the same keys. This is different from “double signing” when you’re a validator signing the same blocks differently (with a modified Harmony core code)
[STABILITY] We must run on HIGH CAPACITY servers
We measured our node performance during Foundational Node days and noticed that the CPU, memory and network demands are low for the most parts, with times when it spikes.
We decided to run on a slightly higher capacity servers according to the recommendations of Harmony. Sure, node operators can run on a single CPU, with 4GB RAM, and a low throughput node, to save costs. But we believe this is bad for the overall network’s Uptime and decide to run on higher capacity servers.
[ALERTING] We must MONITOR the service and the monitor too
“Monitor the monitor” … we have layered alerting at the instance level, the network port levels, and at the consensus level. Multiple layers of monitoring for many use cases from a node being out of sync, to a server disconnected from their peers for one reason or another. These weren’t built overnight and being a Foundational Node operator definitely helped start us off strong.
[AFFINITY] We named it … 🌐 KysenPool Dragon 🐉
There’s a meridan globe which means we’re a global Validator covering multiple timezones. The Dragon was the new suffix describing the long and ferocious history of folks behind Kysen.
🌐 KysenPool.io is our moniker across Cosmos, Terra and XRP
We looked into names of Greek Gods but didn’t see too much of an affiliation with them, and felt a legendary Dragon was a more suitable fit, with some of us having a strong sense of association with the Lunar Calendar system and a Year of the Dragon (2024) is when we’ll see a breakout year for all blockchains adoptions.
A Moment of Appreciation
Our journey a Validator has been a highly adapted learning experience. We learn, we pivot and we execute.
We would like to take a moment to thank all of those that believed in us and decided to stake with our Validator node.
We will not disappoint, we never want to, and we aim to be the best at what we do with the capacity that’s available to us. As we continue to grow and mature, it is our philosophy to promote the ENTIRE validator community as much as we possibly can.
Telegram and Discord can be a huge distraction.
We will pan away every now and then to focus on what’s necessary to build the best for the blockchain community, to help support, promote, run and build the next Decentralized Internet.
Thank you.
Official Channels
Telegram: t.me/KysenPool
Twitter: twitter.com/KysenPool
Medium: medium.com/kysenpool
Facebook: facebook.com/kysenpool
Weibo: weibo.com/KysenPool
WeChat:
