THORChain bare-metal validator — Part 2: Multi-Node using MicroK8s

This guide is the Part 2 of the THORChain bare-metal validators series, see Part 1 about building the hardware for this server.

In this guide, we will go over configuring MicroK8s on a bare-metal server, installing multiple THORChain node, sharing common external chains, and hide them behind VPN Proxy using WireGuard. This configuration is currently used to run multiple active validators on THORChain network.

At time of writing (Q1–2023), it is possible to run very smoothly up to 7 validators node on a bare-metal hardware having 128 Thread CPU, 750 GB of RAM, and 20 TB of storage, as describe in the hardware Part 1.

This guide will also describe how to share chains with other bare-metal server. This can be used to either combine two similar bare-metal server to run up to 15 THORChain validators, or to propagate the workload on multiple less-powerful bare-metal server. For example, one could have a bare-metal dedicated to the chains-daemons, and a bare-metal dedicated to THORCHain validators. While it is possible to group multiple bare-metal together into a Kubernetes Cluster, this may cause more complication than simply sharing the daemon services to the local area network.

In this guide, we will also be hiding our THORChain validators behind a VPN-like proxy server using WireGuard.

This can be useful to hide the real location of a bare-metal server, to run it at a local data center, or even from home, for example, and may prefer the actual location to not be doxed via its IP address.

In addition, this could also help increase resiliency. If a hosting company decides to close the account for non-crypto policy or anything else, we don’t need to rebuild the whole node like if it was running in the cloud, only the Proxy Server would be affected and could be rebuilt anywhere else within minutes.

The WireGuard approach also saves us from the need to have any Static IP at the location of our bare-metal server. One Static IP per Validator would normally be required.

I also want to thank both Hildisvíni Óttar and Scorch who participated in great discussion with me at the time of configuring this server, as well as Multipartite for his great analysis and feedback.

MicroK8s

I chose to give a try to MicroK8s this time instead of K3s, I was not familiar with it but it was highly recommended by a few.

Note: Before installation, it is recommended that the server get a reserved IP from the LAN DHCP server. MicroK8s is using the LAN IP hardcoded in some of its configuration and may give some problem if the it get assigned a different IP from the DHCP server in the future.

Installation:

sudo snap install microk8s --classic --channel=1.28/stable

Update (later):

If later on we want to update, for example from v1.26 to v1.28:

sudo snap refresh microk8s --classic --channel=1.28/stable

Confirm Installation:

sudo microk8s status
sudo microk8s kubectl get nodes

MicroK8s — Add-on

Some add-ons need to be enabled in MicroK8s to support our configuration.

This is not required for additional kube node to a cluster, add-ons will be propagated to all nodes via the cluster.

Enable Add-ons (dns, hostpath-storage, metrics-server):

sudo microk8s enable dns hostpath-storage metrics-server

dns — is required for hostnames between pods, it is important that this is installed before the first namespace/pod is created on the server.
hostpath-storage — is required to put pod storage on host NVMe.
metrics-server — is required to run “kubectl top node”.

Enable Add-ons (MetalLB):

sudo microk8s enable metallb

This will prompt to enter an IP range for MetalLB, we can leave this empty for now, or enter any fake IP as a place holder easier to modify later: 1.2.3.4/32

MetalLb — is required to assign IP to specific nodes.

MicroK8s — Join Cluster (Optional — Additional Cluster Node Only)

This step is only required when combining multiple bare-metal server into a cluster of kubernetes nodes. The whole setup can be followed with only one bare-metal server, and an additional server can be added to the cluster afterward.

Limited use of Clusters — In this setup, because we are using the hostpath-storage to store the data of our THORChain pods, we can’t leverage the High Availability functionality of MicroK8s, but a Cluster can still allow our Pods from multiple bare-metal server to see each other. A ThorChain Node on one bare-metal server could point to chains-daemons running on a different bare-metal server if they are in the same Cluster.

After going the Cluster route, I opted to simply publish the Chains-daemon service to the LAN instead.

Configure Hosts files (Optional)

All Servers need to be reachable from others via their hostnames to be linked via a Cluster.

This can be done bin adding them to the hosts file on all servers.

sudo nano /etc/hosts

Join Existing MicroK8s Cluster

// (On the existing Node)
sudo microk8s add-node
// This will return the command with the Token to be run on the new Node.

// (On the new Node)
microk8s join <IP>:25000/<Token>

// Verify (On the new Node)
sudo microk8s kubectl get node
sudo microk8s status

We can see via the status command that required add-ons were enabled on the new kubernetes node by joining the cluster.

Note: If getting the error “Connection failed. The hostname (xxxx) of the joining node does not resolve to the IP “x.x.x.x”. Refusing join (400).” It means that our hostname can’t be resolved by each other, review hosts files.

Kube Environment Configuration

This configuration allows kube client and tools (such as k9s) to interact with our Kube node/cluster.

# Create Folder if it doesn't exist
mkdir ~/.kube

// Export Kube Config
sudo microk8s config > ./.kube/config

// Edit bashrc
nano .bashrc

Add the following lines in .bashrc and save file:

# Config for K9s
export KUBECONFIG=~/.kube/config
# Use nano instead of vi as default editor
export KUBE_EDITOR="nano"
# Autocomplete kubectl
source <(kubectl completion bash)

// Reload bashrc to apply the changes to the current session
source .bashrc

Kubectl

This allows to use the kubectl command directly without prefixing it with microk8s every time. It is also required to run some of the THORChain scripts which calls kubectl directly.

sudo snap install kubectl --classic

K9s Console (Optional)

K9s is a nice and powerfull console to monitor and interact with the pods of our cluster.

Copy latest Linux x86_64 release path from: https://github.com/derailed/k9s/releases

// Go to Home Directory
cd

// Download
wget https://github.com/derailed/k9s/releases/download/v0.27.3/k9s_Linux_amd64.tar.gz

// Extract
tar -xvzf k9s_Linux_amd64.tar.gz

// Move to Binary folder
sudo mv k9s /bin

// Clean Up
rm LICENSE README.md k9s_Linux_amd64.tar.gz

// Try
k9s
// Ctrl+C to Exit

Troubleshooting

// Get Environment Info (Config and Log files location)
k9s info

Error “ERR refine failed error=”Invalid kubeconfig context detected” indicate that the KUBECONFIG variable was not found.

CoreDNS Crashloop

CoreDNS Add-on of MicroK8S get confused because Ubuntu use systemd-resolved instead of resolvconf as DNS. To resolve this we need to alter the resolv.conf symlink.

sudo mv resolv.conf resolv.conf.originallink

sudo ln -s /run/systemd/resolve/resolv.conf resolv.conf 

cat /etc/resolv.conf

sudo microk8s kubectl rollout restart -n kube-system deployment/coredns

kubectl logs coredns-6f5f9b5d74-2tq95 -n kube-system

StorageClass

StorageClass indicate where Microk8s will store pods storage on our local host, we want to point this on our NVMe Raid.

This is not required for additional kube node to a cluster.

Prepare StorageClass object for NVMe Raid

cd
mkdir mk8sconfig
nano mk8sconfig/nvme-hostpath-sc.yaml

Copy the Following yaml Content:

# nvme-hostpath-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nvme-hostpath
provisioner: microk8s.io/hostpath
reclaimPolicy: Delete
parameters:
  pvDir: /data
volumeBindingMode: WaitForFirstConsumer

Create StorageClass:

// Create StorageClass from yaml file
kubectl apply -f mk8sconfig/nvme-hostpath-sc.yaml

// Confirm creation of new StorageClass
kubectl get storageclass

Set Default StorageClass

// Display StorageClass
kubectl get storageclass

// Set Default
kubectl patch storageclass microk8s-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

kubectl patch storageclass nvme-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

// Confirm the new Default
kubectl get storageclass

Note: If using a multi-node kube cluster, the same storageClass will be used for each node in the cluster, they will all refer to the same path (/data) which will be interpreted locally (each their own /data folder) on whenever server a new pod is created, is it important that each server of the cluster uses the same path for their data folder.

Create THORNode Shared Chains Daemons

The intention of this setup is to be able to run multiple validator nodes on a bare-metal server. To save on resources, we want to be running only one instance of some chain daemons and share it with every node, instead of every node running it own instance of every chain daemons. There is no advantage to run 4 copy of the ETH chain on our server.

This approach is based on the Multi-Validator-Cluster document published by 9R.

For this setup, I chose to share a single daemon instance of all non-UXTO chains with every node, while having individual daemon instances for all UXTO chains. From my other members experience, UXTO chains are less friendly to share between multiple node and may be a cause of slash because they require confirmations.

While I have not tried myself to share UXTO between multiple nodes, I can confirm that this half-and-half setup is working very well. Let me know if you explored other combinations.

Edit: Since the UXTO Client V2, we can now share UXTO Chain with multiple nodes, but they may require some additional memory.

Install ThorChain Node Pre-requirements:

sudo apt-get install make wget jq -y

Prepare git folder:

cd ~
mkdir c0
cd c0
git clone https://gitlab.com/thorchain/devops/node-launcher.git
cd node-launcher
git checkout master
git config pull.rebase true
git config rebase.autoStash true

Install Tools (First Kube Node Only?):

make helm
make helm-plugins
make tools

Verify all pods are healthy:

k9s
// or
kubectl get pods -A

Change Chaosnet Config to Install only Non-UXTO Chains

cd ~/c0/node-launcher

// Edit Daemons Configs
nano thornode-stack/mainnet.yaml

Disable all UXTO Chains by setting the enabled value to false. We will want to run them locally to each node later.

binance-daemon:
  enabled: true
bitcoin-daemon:
  enabled: false
litecoin-daemon:
  enabled: false
bitcoin-cash-daemon:
  enabled: false
ethereum-daemon:
  enabled: true
dogecoin-daemon:
  enabled: false
gaia-daemon:
  enabled: true
avalanche-daemon:
  enabled: true
binance-smart-daemon:
  enabled: false

Run Make Install to Create Daemons

NAME=c0 TYPE=daemons NET=mainnet make install

// Confirm
k9s
// or

kubectl get pods -A

Second Instance of Ethereum-Daemon (Optional)

Given that ETH is very long to sync (almost two weeks, at time of writing this), it may be worth to have a second ETH daemon that is ready in the background. If our first ETH Daemon ever get corrupted, we could just tell all our validators to point to the second ETH instance.

To do so, we can repeat the previous section using a different namespace (c1 for example), and only enable the ethereum-daemon.

ethereum-daemon:
  enabled: true

Shared Daemon with other bare-metal via Kube Cluster

When two bare-metal are linked via a Kubernetes Cluster, pods from both server can see each other and can be reached via their Kube FQDA similar to:

  ethereumDaemon:
     mainnet: http://ethereum-daemon.c0.svc.cluster.local:8545

The c0 represent the namespace the daemon is running in.

If planning to run a multi-node kube cluster, each bare-metal server could have their own set of shared chain-daemons that can be used locally, and be the backup of each other in case of corruption.

Note that running a Kubernetes Cluster add complexity to the setup and may turn out into unexpected behavious, such as pods trying to start on the wrong kube node, etc.

Shared Daemon with other bare-metal via Network (Optional)

A simple way to share daemon with another bare-metal is simply to set an External-IP in the daemon kube service.

In the following example, we want to expose our ethereum-daemon of namespace c0, to be reachable via the LAN IP. This would allos the Eth RPC to be reachable by any computer on the Local Area Network.

kubectl patch svc ethereum-daemon -n c0 -p  '{"spec":{"externalIPs":["192.168.1.50"]}}'

List services to confirm

kubectl get services -n c0

The daemon can then be access by TC Node running on other bare-metal server via the following config:

  ethereumDaemon:
      mainnet: http://192.168.1.50:8545

We can repeat the same with other services as desired, as long as they are exposing unique port number.

There is many other way to expose ports externally, this was the easiest.

Setup WireGuard Proxy Server (one for each validator)

We can use pretty much any inexpensive cloud vm to host a WireGuard Proxy (cloud, shared-hosting, bare-metal, etc), as long as it has a static IP, and a fast network connection, it does not require high resources specs (1 CPU with 500 MB of RAM, and 10 GB Storage is enough). One could also consider it to be located in a country less prone to interventions.

In this guide we are using Ubuntu LTS as an OS for the WireGuard Proxy, similar to this guide.

Update the OS:

sudo apt update
sudo apt upgrade -y

If prompted for merging, select first option, “install the package maintainer’s version”.

Install Networking and Monitoring Tools (Optional)

sudo apt install glances nmap net-tools iperf3 tshark speedtest-cli traceroute -y

Install WireGuard

sudo apt install wireguard -y

Restart to reload services and new kernel if required.

sudo reboot

Generate WireGuard Private and Public KeyPairs

// Generate KeyPair for WireGuard Server (The Proxy itself)
wg genkey | sudo tee /etc/wireguard/wg1server.key
sudo chmod go= /etc/wireguard/wg1server.key
sudo cat /etc/wireguard/wg1server.key | wg pubkey | sudo tee /etc/wireguard/wg1server.pub

// Generate KeyPair for THORChain Node (Acting as WireGuard Client)
wg genkey | sudo tee /etc/wireguard/wg1node.key
sudo chmod go= /etc/wireguard/wg1node.key
sudo cat /etc/wireguard/wg1node.key | wg pubkey | sudo tee /etc/wireguard/wg1node.pub

// Generate KeyPair for Additional Client sur as Laptop or Cellphone.
wg genkey | sudo tee /etc/wireguard/wg1client.key
sudo chmod go= /etc/wireguard/wg1client.key
sudo cat /etc/wireguard/wg1client.key | wg pubkey | sudo tee /etc/wireguard/wg1client.pub

Create Config File for Server

nano /etc/wireguard/wg1.conf

Copy the following Content:

[Interface]
Address = 10.10.1.100/24
PrivateKey = <wg1server.key>
ListenPort = 51820 // Can change that default 
SaveConfig = false

PostUp = ufw route allow in on wg1 out on eth0
PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE

PreUp = iptables -A FORWARD -i eth0 -o wg1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PreUp = iptables -A FORWARD -i wg1 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PreUp = iptables -A FORWARD -i eth0 -o wg1 -p tcp --syn --match multiport --dports 6040,5040,27146,27147 -m conntrack --ctstate NEW -j ACCEPT
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --match multiport --dports 6040,5040,27146,27147 -j DNAT --to-destination 10.10.1.101
PreUp = iptables -t nat -A POSTROUTING -o wg1 -p tcp --match multiport --dports 6040,5040,27146,27147 -d 10.10.1.101 -j SNAT --to-source 10.10.1.100

PreDown = ufw route delete allow in on wg1 out on eth0
PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

PostDown = iptables -D FORWARD -i eth0 -o wg1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -D FORWARD -i wg1 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o wg1 -p tcp --syn --match multiport --dports 6040,5040,27146,27147 -m conntrack --ctstate NEW -j ACCEPT
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --match multiport --dports 6040,5040,27146,27147 -j DNAT --to-destination 10.10.1.101
PostDown = iptables -t nat -D POSTROUTING -o wg1 -p tcp --match multiport --dports 6040,5040,27146,27147 -d 10.10.1.101 -j SNAT --to-source 10.10.1.100

# Node
[Peer]
PublicKey = <wg1node.pub>
AllowedIPs = 10.10.1.101/32

# Laptop
[Peer]
PublicKey = <wg1client.pub>
AllowedIPs = 10.10.1.102/32

Enable IP Foward

// Add the following config at the end of sysctl.conf:
sudo echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf

// Apply Changes
sudo sysctl -p

Run WireGuard Manually (Optional)

sudo wg-quick up wg1
sudo wg-quick down wg1

Set WireGuard as Service

sudo systemctl enable wg-quick@wg1.service
sudo systemctl start wg-quick@wg1.service
sudo systemctl status wg-quick@wg1.service

Watch for Connection (Troubleshooting)

wg show
watch wg

Look for peer latest handshake

Firewall Configuration

sudo ufw reset

sudo ufw default allow incoming
sudo ufw default allow outgoing
sudo ufw allow in on eth0 to any port 22
sudo ufw allow in on eth0 to any port 5040
sudo ufw allow in on eth0 to any port 6040
sudo ufw allow in on eth0 to any port 27146
sudo ufw allow in on eth0 to any port 27147
sudo ufw allow in on eth0 to any port 51820
sudo ufw deny in on eth0

sudo ufw enable
sudo ufw status numbered

Try connecting another SSH Session before closing the current one to confirm that the server can still be connected to.

Test Ports (from a different computer)

nmap -Pn -p 22,23,25,80,1317,8080,5040,6040,26656,26657,27146,27147,51820 <ProxyPublicIP>

After everything is configured the following ports should be Open: 22, 5040, 6040, 27146, 27147, 51820, everything else would be Filtered.

Repeat this part to create a WireGuard Proxy for each Validator Node. I created them as wg1 wg2 wg3, with IP range 10.10.1.x, 10.10.2.x, 10.10.3.x.

Setup WireGuard Proxy Client (one for each node)

Back on the Bare-Metal Server, we will configure WireGuard at that end to connect to the Proxy we just created.

Install WireGuard (one time)

sudo apt install wireguard -y

Install Dependency (one time)

sudo apt install openresolv -y

Create Config files for Server

sudo nano /etc/wireguard/wg1.conf

Copy the following Content:

[Interface]
Address = 10.10.1.101/32
PrivateKey = <wg1node.key>
DNS = 9.9.9.9
SaveConfig = false

[Peer]
PublicKey = <wg1server.pub>
EndPoint = <Proxy Server Public IP>:51820
AllowedIPs = 10.10.1.0/24
PersistentKeepalive = 25

Run WireGuard Manually (Optional)

sudo wg-quick up wg1
sudo wg-quick down wg1

Set WireGuard as Service

sudo systemctl enable wg-quick@wg1.service
sudo systemctl start wg-quick@wg1.service
sudo systemctl status wg-quick@wg1.service

Ping Test

ping 10.10.1.100

Watch for Connection (Troubleshooting)

wg show
watch wg

Setup WireGuard IP in MetalLB

kubectl edit ipaddresspool default-addresspool --namespace=metallb-system

Add the IP to the IP List.

spec:
  addresses:
  - 10.10.1.101/32


(Ctrl-X to Save)

Confirm Change

kubectl describe ipaddresspool default-addresspool --namespace=metallb-system

Repeat for each VPN Tunnel.

Install THORChain Node (one for each validator)

For this setup we will create a distinct git working folder for each validator node.

Prepare git folder:

cd ~
mkdir n1
cd n1
git clone https://gitlab.com/thorchain/devops/node-launcher.git
cd node-launcher
git checkout master
git config pull.rebase true
git config rebase.autoStash true

Change Chaosnet Config to Install only UXTO Chains daemon and point to earlier created Non-UXTO Chains.

cd ~/n1/node-launcher

// Edit Daemons Configs
nano thornode-stack/chaosnet.yaml

Add the following values at the end of the file:

# point bifrost at shared daemons
bifrost:
  binanceDaemon:
    mainnet: http://binance-daemon.c0.svc.cluster.local:27147
# Keep commented out, running the following chain localy 
#  bitcoinDaemon:
#    mainnet: bitcoin-daemon.daemons.svc.cluster.local:8332
#  litecoinDaemon:
#    mainnet: litecoin-daemon.daemons.svc.cluster.local:9332
#  bitcoinCashDaemon:
#    mainnet: bitcoin-cash-daemon.daemons.svc.cluster.local:8332
#  dogecoinDaemon:
#    mainnet: dogecoin-daemon.daemons.svc.cluster.local:22555
  ethereumDaemon:
    # Link Eth RPC from Cluster
    mainnet: http://ethereum-daemon.c0.svc.cluster.local:8545
    # Link Eth RPC from External Cluster
  # mainnet: 192.168.1.50:8545
  gaiaDaemon:
    enabled: true
    mainnet:
      rpc: http://gaia-daemon.c0.svc.cluster.local:26657
      grpc: gaia-daemon.c0.svc.cluster.local:9090
      grpcTLS: false
  avaxDaemon:
    mainnet: http://avalanche-daemon.c0.svc.cluster.local:9650/ext/bc/C/rpc

Disable all non-UXTO Chains by setting the enabled value to false. They are shared between all nodes.

binance-daemon:
  enabled: false
bitcoin-daemon:
  enabled: true
litecoin-daemon:
  enabled: true
bitcoin-cash-daemon:
  enabled: true
ethereum-daemon:
  enabled: false
dogecoin-daemon:
  enabled: true
gaia-daemon:
  enabled: false
avalanche-daemon:
  enabled: false
binance-smart-daemon:
  enabled: false

Set loadBalancerIP parameter for Gateway

This will force MetalLB to assign a specific IP to this validator. We need that so that the node actually receive the same VPN IP than the one we configure internally for it to be reached thought.

nano gateway/templates/service.yaml

Add the following values in the section metadata, annotations:

metadata:
  annotations:
    #MetalLb - WG
    metallb.universe.tf/loadBalancerIPs: 10.10.1.101

Set External Environment IP for Bifrost

This will allow Bifrost to broadcast the proxy public IP as the IP to be reached at from other Validators.

nano bifrost/templates/deployment.yaml

Hardcode IP value in the section Env

          env:
            - name: EXTERNAL_IP
              value: "1.2.3.4"
            #  valueFrom:
            #    configMapKeyRef:
            #      name: {{ include "bifrost.fullname" . }}-external-ip
            #      key: externalIP

Set External Environment IP for Thornode

This will allow Thornode to broadcast the proxy public IP as the IP to be reached at from other Validators.

nano thornode/templates/deployment.yaml

Hardcode IP value in the section Env

          env:
            - name: EXTERNAL_IP
              value: "1.2.3.4"
            #  valueFrom:
            #    configMapKeyRef:
            #      name: {{ include "thornode.fullname" . }}-external-ip
            #      key: externalIP

Run Make Install to Create Node

NAME=n1 TYPE=validator NET=mainnet make install

Confirm Pods are starting correctly

k9s
// or
kubectl get pods -A

Troubleshoot — Pod Pending

If a pod is stuck in Pending state, looking at the description could provide information about what is going on, could be that the host is missing resources to allocate to it.

Troubleshoot — Pod Init

if a pod is stuck in init state, looking at the logs of the init container could indicate the cause of the hang.

kubectl logs thornode-688467cf75-xxxxx -c init-external-ip -n n1

Confirm Gateway was assigned the correct External-IP by MetalLB

kubectl get services --namespace n1 --field-selector metadata.name=gateway

Confirm Bifrost and Thornode External IP Configuration

kubectl exec -i -t deployment/bifrost -n n1 -- printenv |grep IP
kubectl exec -i -t deployment/thornode -n n1 -- printenv |grep IP

Confirm Ports are Open from Proxy Public IP

nc -v <ProxyPublicIP> 6040
curl <ProxyPublicIP> 6040

Review Ports Open from Proxy Public IP

nmap -Pn -p 22,23,25,80,1317,8080,5040,6040,26656,26657,27146,27147,30000-30004 <ProxyPublicIP>

Sync Thornode Chain from Snapshot (Optional)

NAME=n1 TYPE=validator NET=mainnet make recover-ninerealms

// Select Pruned
// Select Highest Block

Wait for all chains to Sync

NAME=n1 TYPE=validator NET=mainnet make status

Finish the Validator Configuration as Usual.

Complete Validator configuration as usual.

The only exception would be when setting the IP Address. Using make set-ip-address will broadcast the wrong IP Address. It is important to broadcast the External IP of the WireGuard Proxy by running the following command instead.

kubectl exec -it -n n1 deploy/thornode -- /kube-scripts/set-ip-address.sh "<ProxyExternalIP>"

Bash Prompt (Optional)

The following bashrc helps navigate through multiple validators.

Edit bashrc

nano ~/.bashrc

Add the following:

alias ms='echo $(date); uptime; apcaccess | grep --color=never  LOADPCT; df -h /data; NET=$NET NAME=$NAME TYPE=$TYPE TC_BACKUP=0  make status'

alias makeinstall='NET=$NET NAME=$NAME TYPE=$TYPE TC_BACKUP=0  make install'
alias makeupdate='NET=$NET NAME=$NAME TYPE=$TYPE TC_BACKUP=0  make update'
alias makeshell='NET=$NET NAME=$NAME TYPE=$TYPE make shell'
alias makelogs='NET=$NET NAME=$NAME TYPE=$TYPE make logs'

export color_chainactive="1;36m"
export color_chainstandby="0;37m"

export color_active="1;35m"
export color_standby="1;33m"
export color_free="1;32m"


alias c0='echo "switch to: Shared Chains 0"; PS1="\[\033[$color_chainactive\]C0-CHAIN\[\033[01;34m\] \w\[\e[m\]\\$ "; NET="mainnet"; TYPE="daemons"; NAME="c0"; cd ~/c0/node-launcher'
alias c1='echo "switch to: Shared Chains 1"; PS1="\[\033[$color_chainstandby\]C1-CHAIN\[\033[01;34m\] \w\[\e[m\]\\$ "; NET="mainnet"; TYPE="daemons"; NAME="c1"; cd ~/c1/node-launcher'

alias n1='echo "switch to: Node 1"; PS1="\[\033[$color_free\]N1-ABCD\[\033[01;34m\] \w\[\e[m\]\\$ "; NET="mainnet"; TYPE="validator"; NAME="n1"; cd ~/n1/node-launcher'
alias n2='echo "switch to: Node 2"; PS1="\[\033[$color_standby\]N2-EFGH\[\033[01;34m\] \w\[\e[m\]\\$ "; NET="mainnet"; TYPE="validator"; NAME="n2"; cd ~/n2/node-launcher'
alias n3='echo "switch to: Node 3"; PS1="\[\033[$color_active\]N3-IJKL\[\033[01;34m\] \w\[\e[m\]\\$ "; NET="mainnet"; TYPE="validator"; NAME="n3"; cd ~/n3/node-launcher'

Reload bashrc

source ~/.bashrc

Ensure important Ports are open prior to churn

Running the following from another computer to confirm that the 4 Ports are open. These are important to establish correct communication to Churn-In.

nmap -Pn -p 5040,6040,27146,27147 <ProxyExternalIP>

Update Node

Update Shared Chains

cd ~/c0/node-launcher
git pull --rebase --autostash
NAME=c0 TYPE=daemons NET=mainnet make install

Update Validators

cd ~/n1/node-launcher
git pull --rebase --autostash
NAME=n1 TYPE=validator NET=mainnet make update

cd ~/n2/node-launcher
git pull --rebase --autostash
NAME=n2 TYPE=validator NET=mainnet make update

Scale Down Pods to Reboot Server

Scaling down Pods prior to rebooting server can help preventing chain corruptions.

// Scale down all Pods
kubectl -n c0 scale deployments --replicas=0 --all
kubectl -n n1 scale deployments --replicas=0 --all
kubectl -n n2 scale deployments --replicas=0 --all
kubectl -n n3 scale deployments --replicas=0 --all

// Wait all pods to terminate
k9s

sudo shutdown -h now

Scale up after server boot up

kubectl -n c0 scale deployments --replicas=1 --all
kubectl -n n1 scale deployments --replicas=1 --all
kubectl -n n2 scale deployments --replicas=1 --all
kubectl -n n3 scale deployments --replicas=1 --all

Conclusion

This guide cover a pretty innovative way to run THORChain Validators with a lot of flexibility. Please reach out to me should you have any suggestions on how to improve this setup.

See Part 2 about adding a MAYA Node to this Setup.

See my other guide: THORChain Validators Migration — The Exhaustive Guide for guidance with migration of an active validator from the cloud to a bare-metal server.