Build your own THORChain Validator Node — Step by step guide

Changes

17 Jan 23 — Clarified External IP settings
18 Jan 23 — Added storage alias command to keep an eye of storage used and actual PV sizes on disk.
11 Jun 23 — K3s compatibility issue added.
25 Aug 23 — Added wireguard guidance, removed ip obfuscation.

If you are reading this you probably know of THORChain, a DeFi project run with its own blockchain network and allows for native layer 1 swaps without the need for vulnerable blockchain bridges or wrapped tokens. It can be done with ease, speed and minimal fees e.g BTC to ETH. This network is maintained by decentralised Validators.

These validators are spread throughout the world and are hosted on either; common cloud providers such as Amazon, Google and Digital Ocean; Bare metal providers such as Vultr or independently like home servers.

For detailed information on what is a Node Operator and how to setup one, refer to the THORChain documents and GrassRoots Videos.

This guide will give you a detailed explanation of everything that is needed to setup and run your own validator and combines what has previously been written by Hildisvíni Óttar.

It should be read in conjuction with the THORChain documents and the existing bare metal blogs written by Hildisvíni Óttar, (VULTR bare-metal THORChain validator setup guide and Bare Metal THORChain validators).

Note: I have not included everything from those blogs. Only major components.

Pre-requisites

1. Knowledge on the THORChain protocol — https://thorchain.org/
2. Dev Op skills — See https://docs.thorchain.org/thornodes/overview
3. Knowledge on building PC’s
4. Basic familiarity on Linux (Ubuntu) and Command line interface commands (CLI)
5. Access to RUNE for bonding (individually or through a pool of people)— Minimum 300,000 Rune, 800,000 to be competitive.
   See Mimir values for minimum requirements: https://midgard.ninerealms.com/v2/thorchain/mimir
   “MINIMUMBONDINRUNE”: 30000000000000
6. A stable, fast internet connection
7. Stable power source

Knowledge on THORChain

There is heaps of information on what THORChain is and the roles validators have.
Here are a few resources to brush up on if you are still unsure

A. THORChain website — https://thorchain.org/

B. THORChain online docs — https://docs.thorchain.org/

C. Discord Channel — https://discord.com/invite/KjPVnGy5jR

D. Grassroots videos — https://www.youtube.com/@GrassRootsCrypto

E. THORChain blog site — https://medium.com/thorchain

Step 1 — Home preparation

1. Internet — Ensure you upgrade or have an existing high speed connection aroung 1GBps .
2. Also ensure your internet plan has a Static IP upgrade and make sure your connection is not behind Carrier Grade NAT (CGNAT) (call your ISP and have it disabled. Some ISPs offer gaming plans which come with Static IPs and are not behing their CGNAT.
3. Solid power connection — For this I rely heavily on the main grid, but supplement it with home solar and a UPS (capable of running the system for an hr).
   I connected an energy cost meter to my UPS which has the PC and Internet connected and it is rougly using 180w +/-10w. This works out to be about $1USD a day in power before taking into account solar.

Note: 99% uptime per year still allows for 3–4 days of power outages accumulated through the year so a UPS is not necessary. If you are happy to be slashed during outages then you don’t need it, however, it does also add surge protection with an equipment gurantee.

6. Wifi/internet Backup. Again, not mandatory, but if you live in an area with poor internet reliability you may consider buying a $50 wifi dongle and using it in an standby mode. To do this you will need to do a network bond (different from Rune bonding) which adds an Active-Backup policy to your internet connection so if your main connections disconnects, the backup will start to kickin. I didn’t do this, because where I live, if the power goes down, the phone reception becomes poor, so a wifi mobile phone backup would be too slow and unstable to maintain a connection.

If you want to do this, see this resource for more information:
https://techviewleo.com/how-to-configure-network-bonding-on-ubuntu/

Security

To reiterate what Hildisvíni Óttar said, ensure you have adequate physical sercurity and computer security.
If you are running your server from home you should setup IP Anonyminity.
I explain how to achieve this later on set by step.

Step 2 — System build

You need a suitablly fast and powerful computer to be able to handle all the data coming in to the cluster 24–7. For future proofing (when additional chains are added we need to make sure what we purchase will suffice for the next few years. You also need significant Ram for caching large chain files. Remember, building and running a validator is serious business, so dont skimp on computer parts or you will regret it when it gets slashed all the time or cannot handle network expansion in the future.

I took Hildisvíni Óttar’s advice and opted for a 16 core CPU Ryzen 3 with 8TB of M.2 SSD storage and 128GB of RAM. I also added an old graphics card I had at home to make setup easier. If you dont have a spare one, you can get a super cheap one as you are only using it to setup the system. My exact build is here:

https://pcpartpicker.com/list/Lb78k9

My power supply is over the top so feel free to change this.

NOTE: Dont skimp on hard drives! We need our hard drives to have fast read and write speeds and M.2 SSDs are the fastest! You can check the warranty and specifications/requirements to make sure the SSDs can handle the data writing long term . The thing to check for is Terabytes Written (TBW)(Max Capacity)*. This is a warranty related number. A smaller number means if you go over the total bytes written number inside your warranty period and the drive fails, it is void). The ones I chose have a TBW of 2560. Others can be much lower (around 800TBW).
NOTE:Once the system is built you can monitor it using SMART status.

If you don’t have the knowledge to build the computer yourself, use a computer company in your area, so you don’t damage or stuff up the build.

Once the system is built we can move onto installing Ubuntu and getting our partitions right.

Step 3 — Install and setup

The trickest bit in the installation is setting up the M.2 SSDs to be a single bootable logical drive. We want to make a single volume as part of the OS install. This will lower your admin exponentially by having everything on one single disk allowing the cluster to grow as needed and without having to check disk sizes, relocating pods or customize the install/upgrades etc as THORChain grows and gets updated.

1. Download Ubuntu and create a bootable USB stick.
2. Click install
3. Connect to wifi to get the latest drivers
4. Select minimal + download updates and (optional) — install 3rd party software for graphics and wifi
5. Erase and Install Ubuntu
6. Advanced
7. Select Use LVM

Click through the remaining steps and let Ubuntu install. Don’t be worried if it says it will take up the entire disk. We want this.

Ensure you give your username a long password for security reasons.

Once it has installed and you have rebooted your machine, login and move to the next step.

Now that we have a clean Ubuntu installation we need to add the second HDD to the primary storage volume know as root (mount point is / ).

To understand how Ubuntu’s LVM works reference the picture below. Also watch the video by Chris Titus here: https://christitus.com/lvm-guide/

In our setup we have 2 M.2 SSDs (Hard drives). Each has their own partition and physical volume.
When we installed Ubuntu using LVM, it installed a couple of additional smaller partitions (/boot and /efi) on the first drive with their own volumes which get reserved for the system. This means our first hard drive actually has 3 partitions on it. LVM should have also created a volume group and a logical volume where it installed the main system files. Only 1 of the partitions should be part of the volume groupd (/root).

The second SSD should be untouched.

Add new partition to second hard drive

We will take this second SSD, create a partition and physical volume for the entire hard drive, then add it to the same volume group and logical volume that the system created during install.

Open the terminal and install lvm

sudo apt install lvm2

Check your current partitions

df -h

It should look like this

Notice we can only see a 3.4TB volume called /dev/nvme0n1p3 mounted at / ? This is our root volume we are going to add our second hard drive to. Also notice the other 4TB hard drive is no where to be seen. This is because it has neither a physical volume nor belongs to a volume group with a logical volume.

We should now check out our partitions and volumes to see if the second hdd has a partition on it

sudo lsblk

In this case it looks like it has a partition (nvme1n1 with partition nvme1n1p1)

If your second hdd hasn’t been partitioned yet follow these steps.

Go into applications menu à Utilities à Disks
Select the second HDD without no existing partitions
Create a new partition with the entire space
Give it a name like `home` and Type `EXT4`

Now if we run sudo lsblk again we should see the partition under the hdd.

Add a physical volume

Now we will create the physical volume for the second hard drive. First see if one exists for the second hard drive.

Type

sudo pvscan

This should show us if we have one or 2 physical volumes with only one in a volume group.

This is what we are trying to achieve:

We need to add the a physical volume to the second ssd partition before it can be added to the same volume group.

Type df -h to note the file system of the second hard drive. It should be something like /dev/nvme1n1p1

Warning: Creating a pv will wipe all data on it. Make sure you select the correct partition.

Type:

sudo pvcreate /dev/nvme1n1p1

Add new physical volume to existing volume group

Lets get the name of our volume group to add this new volume to.

Type sudo vgdisplay or sudo vgscan to get the volume {name}. Mine was called vgubuntu

Lets extend the volume group to include our new partition. {name} should be whatever the name of the volume group found from above. /dev/nvme1n1p1 is the new partition we created.

sudo vgextend {name} /dev/nvme1n1p1

Now run sudo vgdisplay. It should show us that we have 3.6 TB Free. This means our volume has spare capacity that can be now allocated to new or existing logical volumes.

Extend space in logical volume

Lets add it to our root logical volume.

Run sudo lvdisplay to get the file system location of your existing logical volume. Mine was /dev/mapper/vgubuntu-root.

We will now extend our logical volume to include the extra space that we just added to the volume group.

lvextend -l +100%FREE /dev/mapper/vgubuntu-root– This command is used to extend the logical volume to take up ALL of the remaining free space on the volume group.

Run sudo df -h to check it’s worked. If the /size still says 3.4T it means we need to enlarge the file system allocation.

Type sudo vgdisplay {name} again substitute {name} for your volume group name.

We need to use resize2fs to resize our ext4 file system. This can be used to enlarge or shrink an unmounted file system located on the device.

Run sudo resize2fs /dev/{name}/root it should look like this:

sudo resize2fs /dev/vgubuntu/root

Now running sudo df -h should show the size of / (root volume) to be at 7.2TB

Step 4 — Node preparation

Now that we have our partition all working, we need to install all the necessary programs to get our node launcher running and working properly.

First update the machine and install some necessary programs to work with:

sudo apt-get update && apt-get -y upgrade

sudo apt-get install make wget jq curl git 

Optional: You can also install Smartmontools to check your hard drive health

sudo apt install smartmontools

To check hard drive health you can run: sudo smartctl -A /dev/nmve0

Lets update our shell to zsh

sudo apt install zsh

Now to make zsh more user friendly we can install Oh my zsh (https://ohmyz.sh/). This will allow us to customise the feel of our command line so we can recognise when we have logged into our server.

NOTE: we are directly installing this package from the URL which can be unsafe unless you can trust the url. Feel free to check the URL to ensure it is going to pull the correct package.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

sudo usermod -s /usr/bin/zsh $(whoami)

Install Kubernetes (Rancher K3s)

Lets install K3s

K3s

This needs to be installed without traefik (a reverse http service for microservices). As per Hildisvíni Óttar, you want to install it without traefik using otherwise it hogs port 8080 which Midgard needs and the thornode svclb can’t start, blocking everything else.

Read the compatibility issue below before deciding to proceed with this command.

curl -sfL https://get.k3s.io | sh -s - --disable=traefik --write-kubeconfig-mode 644

Compatibility issue — Jun 23

If installing using the default command above you find bifrost, thornode and gateway pods fail to come online, you should check the logs of gateway.

If it says something like

“Waiting for load balancer external endpoint…”

then its the probably a compatibilty issue with the internal K3 load balancer. Using an older version of k3s works. e.g version 1.25.6+k3s1

To install an old version add INSTALL_K3S_VERSION= into the install line.

curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.25.6+k3s1 sh -s - --disable=traefik --write-kubeconfig-mode 644

Check its installed with kubectl version

Update theme and node alias for `make status`

Lets edit the zsh config file

nano ~/.zshrc

Scroll down and change ZSH_THEME =”robbyrussell” to a name of your choice e.g bob:

ZSH_THEME="bob"

Add the following under where you changed the zsh theme:

alias ms="NET=mainnet NAME=thornode TC_BACKUP=0 TYPE=validator make status"

export KUBE_EDITOR="nano"

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

cd ~/node-launcher

This adds an alias to the Thornode script make status so instead of having to type make status then selecting validator, mainnet, etc you only need to type ms.

Save and exit.

Reload the file:

source ~/.zshrc

Note: You might get an error saying no such file or directory. This is because we haven’t cloned node launcher yet so the command is actually invalid for now.

Customise theme

Lets customise the theme “bob”.

cd ~/.oh-my-zsh/themes


cp robbyrussell.zsh-theme bob.zsh-theme


nano bob.zsh-theme // Change the first line to:PROMPT="%{$fg_bold[white]%}THORNODE %(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ )"$ source ~/.zshrc

Now we have a cool looking prompt that will be obvious when you are logged into your THORNode so you don’t accidentally do destructive commands on it.

Install k9s console

Lets install K9s to give us a node monitoring tool.

Change to root directory (out of node-launcer)

cd ~

Copy the URL of the latest release from: https://github.com/derailed/k9s/releases

At the time of writing it was v0.26.7.
The downloaded file will be called something like `k9s_Linux_x86_64.tar.gz`

Run wget {URL of Linux release}

wget https://github.com/derailed/k9s/releases/download/v0.26.7/k9s_Linux_x86_64.tar.gz

Remove the unnecessary files (readme and license) by doing the following:

tar -xvzf k9s_Linux_x86_64.tar.gz

rm LICENSE README.md k9s_Linux_x86_64.tar.gz

We can check to see if it works:
./k9s will load K9s.
Ctrl + C to exit.

Node launcher install

Make sure you are at root directory ( cd ~) then clone the git directory.

cd ~

git clone https://gitlab.com/thorchain/devops/node-launcher.git

cd node-launcher

git checkout master

Disable CPU mitigations

Lets follow Hildisvíni Óttar’s advice and get our raw CPU performance back and adjust swappiness.

These protections are only for cpu vulnerabilities that exist where running untrusted code on your CPU cores. This would be applicable for shared hosting, performing web browsing etc. Since we’re running a bare bones install on a single tenant hardware node, we want our raw CPU performance back:

Reference: https://sleeplessbeastie.eu/2020/03/27/how-to-disable-mitigations-for-cpu-vulnerabilities/

 lscpu
 sudo nano /etc/default/grub

Add mitigations=off to GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash mitigations=off”

Save, exit and then reload the file:

 sudo update-grub

Now lets reboot and confirm mitigations are offbefore continuing.

 sudo reboot

Once rebooted lscpu and check mitigations are off.

Swappiness

If your setup came with swap enabled, it’s probably set fairly aggressively (default is 60) and isn’t really required with a high RAM server. To turn it down:

cat /proc/sys/vm/swappiness
sudo sysctl -w vm.swappiness=20

Note: Hildisvíni Óttar suggests changing swappiness to 10. This is a personal choice. Choose either 10 or 20.

To make the change permanent:

Edit /etc/sysctl.conf as root

sudo nano /etc/sysctl.conf

Add the following line to the file:

vm.swappiness = 20

Save and exit.

The changes made to sysctl.conf will apply on reboot or you can reload them using sudo sysctl -p

Firewall

Now we will setup the ufw firewall that comes with ubuntu.

We need to ensure all the thornode ports are open and everything else is off.

First lets get the name of the interface that has your ip address

ip address show

Note the name of this device enp0s3.

Now add ufw rules for this device only by typing the following:
Note: it must be done in this order

 sudo ufw reset
 sudo ufw default allow incoming
 sudo ufw default allow outgoing
 sudo ufw allow in on enp0s3 to any port 22
 sudo ufw allow in on enp0s3 to any port 8080
 sudo ufw allow in on enp0s3 to any port 5040
 sudo ufw allow in on enp0s3 to any port 6040
 sudo ufw allow in on enp0s3 to any port 26656
 sudo ufw allow in on enp0s3 to any port 26657
 sudo ufw allow in on enp0s3 to any port 27146
 sudo ufw allow in on enp0s3 to any port 27147

 sudo ufw deny in on enp0s3
 sudo ufw enable

 sudo ufw status numbered

Step 5 — (Optional) — Set up remote access

If you plan on accessing you server remotely you will need to setup ssh.

Create your ssh keygen

Resources: https://www.ssh.com/academy/ssh/keygen
or https://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/javaservice/JCS/JCS_SSH/create_sshkey.html

Install and enable SSH on your server

By default, SSH should already be installed on your host, even for minimal configurations.

To check that this is actually the case, you can run the “ssh” command with the “-V” option.

ssh -V

devconnected.com

If not we can always install it.

sudo apt-get update

sudo apt-get install openssh-server

Check that it is actually up and running.

sudo systemctl status sshd

devconnected.com

By default, your SSH server is listening on port 22.

To check whether your service is enable or not, you can run the following command

sudo systemctl list-unit-files | grep enabled | grep ssh

If you have no results on your terminal, you should “enable” the service in order for it to be launched at boot time.

sudo systemctl enable ssh

By default, SSH configuration files are located in the /etc/ssh folder.

User access

By default, since we are on a new machine your user will be part of the sudo group, but you can check with following.

groups

If for some reason you aren’t you can add your user (bob) to the sudo groups using usermode -aG sudo bob.

Optional: give bob access to sudo without password:

nano /etc/sudoers

Add the following line near the end

bob ALL=(ALL) NOPASSWD: ALL

Give bob your ssh public key:

su bob
mkdir ~/.ssh
nano ~/.ssh/authorized_keys

Paste in your public key which you created earlier from keygen. Save then exit.

Disable root login. Disable password logins

Enter the config file.

nano /etc/ssh/sshd_config

Change to:

PasswordAuthentication no
PermitRootLogin no

Also unhash AuthorizedKeysFile.

Save, exit and restart the service

service sshd restart

Local machine configuration

Configure your local machine to auto-login using namethornode

On the machine you plan on using to remotely login type the following:

nano ~/.ssh/config

Add the following lines:

host thornode-demo
user bob
hostname XX.YYY.ZZZ.AAA

Note: your host name is your IP address. If you are only going to connect from your local network you can use your local internal facing IP address (192.168.X.Y), however if you are going to connect externally you need to set this to your public IP addrtess. If you don’t know what your external ip address is, try typing curl ifconfig.co or googling Whats my ip.

Save and exit.

Check its working (from local machine):

ssh thornode-demo

Step 6 — Router configurations

You may need to configure your Router to open the same ports as the UFW.

You should also configure your server to keep a static IP so if your router is turned off/updated, it doesn’t get reset by automatic DHCP rules and ruin your router port forwarding. This could result in your node being denied the correct connections and you getting slashed.

Go into IPv4 and manually change the method that sets your IP address (e.g 192.168.X.X, netmask and gatway. Gateway can be found in your router settings (external IP) or by googling `my Ip address`. Remember this needs to be a static IP from your ISP.

Check the firewall is working from an external computer:

[your laptop] $ nmap -Pn -p 22,80,8080,5040,6040,26656,26657,27146,27147,30000-30004 <NODE IP>

Step 7 — THORChain installation

Finally we have setup our machine and are ready to install THORChain.

Finally we arrive at the official docs: https://docs.thorchain.org/thornodes/deploying

Install gnu awk.

sudo apt-get install gawk

Then install the support tools for the Node as per Thornode docs.

make helm

make helm-plugins

make tools

Step 7A — IP Anonymity — Wireguard tunnels

WARNING READ BEFORE CONTINUING
If you plan on obfuscating your IP address it is best to setup a wireguard connection (VPN tunnel) that redirects traffic from a public IP to your real IP.

There are several guides on how to do this.
Read the following guides to setup wireguard for your Node.

Bare-metal node: the comprehensive n00b guide

Bare-metal Node: n00b Guide

THORChain bare-metal validator — Part 2: Multi-Node using MicroK8s

THORChain bare-metal validator — Part 2: Multi-Node using MicroK8s

How To Set Up WireGuard on Ubuntu 20.04

How To Set Up WireGuard on Ubuntu 20.04 | DigitalOcean

Set up WireGuard VPN on Ubuntu 20.04

Set up WireGuard VPN on Ubuntu 20.04

(OPTIONAL WAY) Inputting external Wireguard IP into node

As an alternative method to the guides above, you can ensure your external IP is being sent by doing the following:

From your working node-launcher directory

cd bifrost/templates

nano configmap.yaml

Add the following line external_ip="X.X.X.X" near the end of the document where x.x.x.x is your random ip address you found earlier.

It should look like this:

external_ip=”123.456.789.012”

kubectl create configmap $CONFIGMAP --from-literal=externalIP=$external_ip --dry-run=client -o yaml | kubectl apply -f -

Note: Make sure it is tabbed at the same indentation as above. If not you will get an error.

Save and exit.

Now do the same for the directory thornode/templates and nano configmap.yaml. This will be just above the auto-update.sh line.

Remember X.X.X.X is the Wireguard IP address you setup earlier.

Now lets make sure git can accept our changes for future updates.

git config pull.rebase true

git config rebase.autoStash true

Once we run install, Kubernetes should pull this information in.

Now we are ready to install.

Step 8 — Make install

Lets install the node.

make install

Note: Pressing enter to use the default namespace (thornode or thornode-testnet)

You want validator which is All The Chains.

Choose a LONG password.

Once it is installed, wait a few minutes and try and run.

ms or make status

Cluster issues

If ms says its not running, there could be a few pods not working.

Run make pods to verify what is working or not. The most likely problem is the Bifrost or Thornode pods.

You can also check through K9s

cd 

./k9s

View the logs of the orange/non running pods to see where the issues are.

If you have a Kubernetes pod issue. Try restarting this `CTRL + D`. The kubernetes cluster needs to be happy before the other pods can be happy.

Once the cluster is happy, wait another few minutes. Then try restarting thornode and bifrost pods before trying to restart chain pods.

ONLY RESET ONE POD AT A TIME. After each one, wait a few minutes and check the logs.

Once it starts syncing you can check the status using ms or make status.

Sync should take a few days. Some chains are much faster than others.

To speed up the sycn for Thornode run:

make recover-ninerealms

Choose pruned and select the latest block.

Once all chains have sync’d you can follow the thorchain Docs to whitelist your node (send a small amount of Rune to node to register the Node operator account and get whitelisted before continuing.

Manually send IP if make set-ip fails to work

If its showing an internal IP address we will have to manually send it.

Check to ensure the Wireguard IP is configured correctly in the current kube files.

kubectl edit configmap bifrost-scripts -n thornode

kubectl edit configmap thornode-scripts -n thornode

Restart thornode and bifrost make restart.

You can also check bifrost and thornode

make shell [bifrost] and [thornode]

Printenv |grep IP

This should be your Wireguard public IP.

If everything is printing your Wireguard IP you can send it to the network. Replace X.X.X.X with your Wireguard IP

kubectl exec -it -n thornode deploy/thornode -- /kube-scripts/set-ip-address.sh X.X.X.X 

Step 9 (Optional) — Add storage command to easily check storage space and PV cluster sizes.

As your node stays operational, chain pods will continue to grow.
To make sure you keep on top of your storage space we can add an Alias command.

nano ~/.zshrc

Add the following line under where we entered our first alias

alias storage="df -h |grep /dev; sudo du -h -d 1 /var/lib/rancher/k3s/storage"

Save and exit.

Now reload the file

source ~./zshrc

Try it out by typing storage

Step 10 — Bios update

Edit bios to reboot after power loss so your machine spends minimal time offline during power outages.

Step 11- Get node ready for churn in

Follow the THORChain documents to set node keys, version and final bond.

Now you are ready to be an active Validator.

P.S. If something is missing from my guide, please let me know in the THORChain Discord #bare-metal-nodes channel.

Tips welcome

thor17ekvgt4jrrdcq4u0th33rlwy7mfxu360fampyy