Agoric Validator Setup Guide by Chainode Tech
Zug, Switzerland, March 30th 2021 — We are extremely happy to share with our community that Chainode Tech was selected among the 150 Validators that were carefully chosen to join the Agoric Testnet! We have been following the progress of the project for many months now, and we actually decided to join their Testnet even before the incentivised phase began earlier this month.
We would like to get into the details of what setting up a Validator on the Agoric network looks like, what it requires in terms of hardware and supporting tools and what users should be careful about with regard to security and monitoring of the Validator. This being said, let’s get into it!
Validator Infrastructure
The Validator Infrastructure represents the fundament of every decentralized blockchain network as it ensures the well functioning of the network even under high constraints, when the network is being broadly used. Therefore, it is extremely important that validators are prepared and ready to step in during a network overload situation.
We, at Chainode Tech, always use at least 2 times the recommended specifications for our Validator in order to be sure that it will perform without any problems even when the network is being widely adopted and thus more intensively used. Also, we have backup machines ready to step in if the main Validator encounters any issues. In order to improve the liveness, we are typically considering spreading our nodes across multiple geographical regions and we also verify beforehand that we don’t contract the same provider that is used by the majority of the validators in order to increase the security of that specific network.
Agoric Validator Machine Specifications
- 6 CPU Cores AMD EPYC
- 16GB RAM
- 400GB SSD
- OS: Ubuntu 20.04 LTS
Validator Hardening/ Validator Security
For securing and hardening a validator, we released a guide already towards the end of 2019 where we explained that we use HSM modules with 2FA in order to access our validator machines. As an HSM module we are using Yubikey 5Ci with multiple backups so that in case one of them gets broken we would still have access to our validators.
The guide is strictly focused on creating a secure access to your Validator machine and doesn’t describe the firewall setup as well as various key handling.
To this guide we would like to add that it is very important to let only the necessary ports open for communication and restrict RPC communication or allow it only for certain trusted services. Depending on the setup and the cloud providers, we use firewall services and/or we configure ufw or iptables to make sure only necessary ports are open.
Ports:
22 -> standard port open for SSH. In order reduce the number of brute force attacks, you can configure another port for SSH access and combine it with Fail2Ban for more protection;
26656 -> p2p port, open for p2p communication between nodes;
26657 -> RPC port, open only locally and for certain IPs with whom an operator wants to share access;
9464 -> Agoric VM (SwingSet) metrics, open only locally and for certain IPs with whom an operator wants to share access;
1317 -> Cosmos SDK metrics, open only locally and for certain IPs with whom an operator wants to share access;
26660 -> Tendermint metrics, open only locally and for certain IPs with whom an operator wants to share access;
If you plan to use Prometheus and Grafana for monitoring your validator, make sure to also properly define firewall rules and restrict the access for those ports. The standard port for Prometheus is 9090 and for Grafana is 3000 but these can be changed as well.
Another important aspect to consider for validator security is how you handle your keys. As a general rule of thumb it is recommended not to keep more keys than necessary on your validator. For the keys that you need to keep on your validator, it is best if you can use an HSM device or to keep them in the ramfs and restore them from an encrypted cold storage or vault when needed.
Agoric Validator Setup
For creating our Agoric Validator, we used the official guide which we believe offers a very good insight in how the Agoric node and Validator can be created.
It is important to mention that one should start first to sync the node and only when the node has caught up, then the validator should be created. The reason behind this is that once the validator is created it will try to create and sign blocks, although your node isn’t fully synced. Depending on the network parameters, if you create your validator prior to having your node fully synced, it could lead to your validator being jailed.
You can simply check if your node is fully synced by running the command below:
ag-cosmos-helper status
If the node has fully caught up, it will show catching_up:false. Otherwise, if it is still syncing, it will show true for this parameter. Once the parameter will show false, you can proceed with the “Creating the validator” section in the official guide.
In order to expose validator metrics to Prometheus, the following guide is recommended: https://github.com/Agoric/agoric-sdk/blob/master/packages/cosmic-swingset/README-telemetry.md
One thing that we would recommend extra to the official guides is setting a minimum gas price in the app.toml in order to prevent your node from being spammed with transactions, thus influencing the performance of the network and possibly bringing it down.
minimum-gas-prices = “0.025uagstake”
Important mention
It is highly recommended to backup the priv_validator_key.json under .ag-chain-cosmos/config/. This key is the private validator key and if you will lose this and, for any reason, your current validator machine gets destroyed or files are accidentally destroyed, you won’t be able to restore your validator on an existing chain, e.g. agorictest-7 Agoric Testnet. Therefore, we highly recommend to backup this file and store it securely on an encrypted cold storage or vault, especially if it is a key for a Mainnet validator.
Monitoring and Alerting
For Monitoring and Alerting we are using Grafana and Prometheus for monitoring the validator state as well as the validator hardware state, as it can be seen in the screenshots for our Agoric Validator on Testnet. Also, we use certain parameters and criteria in order to send alerts to a dedicated monitoring email address, for example to alert us if the disk space utilization is higher than 70%.
Example: One validator is down — the status for relevant parameters changes and is reflected in the colour as well.
For setting up the Monitoring of the Agoric Validator, the following github repository can be used: https://github.com/Chainode/AgoricTools.
The repository contains guides to install Prometheus, node exporter, Grafana and Grafana plugins as well as a Grafana dashboard to import and monitor your Agoric Validator.
About Agoric
Backed by more than 30 years of experience, the Agoric team is developing a a safer, simpler way to program smart contracts, being built on the Cosmos SDK that uses Tendermint Proof-of-Stake (PoS) and a native token to secure the network. The Agoric chain enables developers to create decentralized applications (dApps) using composable JavaScript smart contracts.
If you are interested to find out more about Agoric, make sure to follow their Twitter, Telegram, Discord and Medium channels.
About Chainode Tech
Founded in early 2019, Chainode Tech is a young, extremely motivated and experienced team based in Zug, Switzerland, that is focused on providing staking and validator services on cutting edge Distributed Ledger Technology (DLT) protocols.
In addition to Agoric, Chainode Tech is active as a Validator also on multiple other Testnet and Mainnet networks, such as Solana, The Graph, Harmony Protocol, ICON Network, KAVA, Akash or CertiK.
