This guide is meant to be an add-on to what has previously been nicely written by Hildisvíni Óttar and Scorch about setting up a bare-metal validator for THORChain.
I’m writing this guide to help the THORChain validators community migrate to bare-metal servers and lower dependency on a few major cloud hosting providers.
In this Part 1, I will be discussing the hardware aspect of my setup. Additional Parts will elaborate more on the THORChain node setup and network configurations.
References
To facilitate readability, I didn’t elaborate on the explanation of steps that were taken directly from one of these previous guides:
- VULTR bare-metal THORChain validator setup guide by Hildisvíni Óttar
A great guide about configuring a Bare-Metal node on Vultr. - Build your own THORChain Validator Node — Step by step guide by Scorch
Another guide about building a Bare-Metal node on your own hardware.
Bare Metal Objective
For this build, I have chosen to go with my own hardware, which is similar to the approach token by Scorch. Hosted bare-metal services such as Vultr are great, but sometime specifications available don’t suit your needs, or their prices end up being up there with pure cloud services such as AWS and DO. Having your own hardware also allows you to easily add additional NVMe drives if required.
The main goal of going Bare-metal is to get away from major cloud hosting services, to ensure THORChain is independent from AWS and DO.
While most would consider going directly to an enterprise-grade server, they are usually very loud, dissipate a lot of heat, amd may also be harder to upgrade. My objective here was to build a setup that was similar to an enterprise-grade server, that can be run in your basement without feeling like you live beside an airport takeoff runway.
Hardware
- Motherboard — Supermicro MBD-H12SSL
This SP3 Socket motherboard supports AMD Gen2 and Gen3 CPUs (EPYC 7003/7002, Threadripper, Ryzen). It also has 8 slots for RAM supporting up to 2TB. The best part of it, it is an ATX form factor, so it can be mounted in a normal consumer PC Case. - CPU — AMD EPYC 7713 (7003 series)
I chose to go with the EPYC Gen3 series because it has more cache than other AMD CPU Series. The EPYC 7713P has 64 cores which is enough to support multiple TC validators. The EPYC 7313P (16 cores) would be sufficient to run a single node with room for additional chains, but I would still recommend aiming for the EPYC 7334P (24 cores) for a single node as it offers more room for future expansion of additional chains. A Ryzen 5950X (16 cores) could also be a good cost efficient option. I used the EPYC 7713P (64 cores) in my latest build, to run multiple nodes on one server, I’ll share more on this once it’s online. - RAM — 1TB DDR4–3200 PC4–25600 RDIMM ECC Registered
I went all out and got the best/fastest server-grade RAM I could find. A single node does run on 128 GB and could be easily upgradable later. - CPU Cooler — Noctua NH-U14S
This CPU Cooler is ultra-quiet and will do a great job even with a very powerful CPU such as the EPYC 7713P. - Power Supply — Seasonic GX-850
A reliable, silent, and modular power supply. It also provides two 8-pin CPU Power Connectors if required. You would be fine with the GM-650. - Case — Antec P101 Silent
This case nicely accommodates our ATX motherboard, has plenty of space for everything, great airflow, and is completely silent. - NVMe — SAMSUNG 990 PRO M.2 2TB
The 990 PRO are amongst the quickest NVMe available on the market to date. Used twelve (12) 2TB (for a total of 24TB) in Raid-0 configuration for a Multiple Node Setup. Could used four (4) of them (for a total of 8TB) for a single node setup. You could cut some expenses on this build by opting for regular M.2 NVMe, especially if you stripe data across many drives (more on this later). Speed and TBW (Total Bytes Written) are probably the most important specs when choosing an NVMe for a THORChain server. - M.2 Expansion — ASUS Hyper M.2 X16 Gen 4 Card
This expansion card allows connecting up to four additional M.2 NVMe on a PCIe 4.0 X16 Slot. It has a built-in heat sink and fan to keep heat away from NVMe, and is pretty affordable. - OS SSD — Kinston DC500R
I chose to get a single SSD dedicated to the Operating System. This is a data center grade one, but any SSD would do. This simplifies the installation compared to Scorch’s guide (see step 3), removing the need to make the NVMe volume bootable. With this setup, the NVMe only stores the Node Pods Data while all the configs are on the OS SSD. This would also allow to replacing/changing the NVMe storage without affecting the main OS. - UPS — APC BN1500M2
This is an affordable consumer grade 900W Uninterruptible Power Supply (UPS). Power outages are not common over here. I use it mainly to protect from power surges, or very short power interruptions. It can still keep my server up for about 40 min if required. It integrates nicely with Ubuntu. The battery can also be replaced easily when required after a few years.
As you can see, I didn’t cut any corners in building this server, and got the best components I could find, and kept room for expansion as the number of supported chains grows over time. The cost of this server is still nothing compared to the AWS Cloud bills.
Assembly
- EPYC Installation Torque: The AMD EPYC CPUs don’t come with the special self-torque Torx screwdriver tool. You can use the same one as the Ryzen CPU if you have one. Otherwise get yourself a Torque Screwdriver that can do 1.6nm (or 14.2 in-lbs), with a T-20 bit. Be very cautious with this step as over-tightening the CPU may damage the CPU and Motherboard.
- Supermicro unique password: The supermicro motherboard comes with a unique password code that you can find on the CPU plastic protective cover, don’t simply discard it. This password is required to connect to IPMI (Supermicro Intelligent Management), the BMC (Baseboard Management Controller Software) of the motherboard, a web front end to manage and monitor the server, similar to the iDRAC for Dell PowerEdge.
- CPU Cooler Installation: Tighten the CPU Cooler screw until they reach the end of the thread, don’t over-tighten them from that point as it could damage the CPU and Motherboard.
- CPU 8-pin Power Connector: This motherboard has two connectors for 8-pin CPU power, with a sticker stating that “Both 8-pin required for heavy load configuration”, to this day I still didn’t figure out what defined a “heavy load configuration”, plugging both for a EPYC 7713, but for an EPYC 7313P, it’s totally fine with using just one.
- BIOS configuration for ASUS Hyper M.2 X16 Gen 4 Card: The X16 bifurcation of this card doesn’t work well with auto configuration and required to go configure it manually. You can do so in the BIOS under: Advanced, PCIe/PCI/PnP Configuration, CPU SLOT<i> PCI-E 4.0 X16 Bifurcation, switch from Auto to 4x4x4x4.
- Identify NVMe disk on Expansion Card: If a NVMe drive ever fail, good chances that we will know the serial number of the faulty drive. If we have multiple Expansion Card it can be hard to know which one contains the faulty drive. I do write the last 4 digit of each NVMe serial number on the Expansion Card Heatsync with a PTouch so that if I ever have a failure I know right away on what card is the faulty device.
Power-on
I told you, this build is silent, I really mean, SILENT. The first time it’s powered on, you may think something is wrong, because there is no sound at all. Just give it a minute or two and BIOS splash screen should display on the screen.
Take a look at the bottom right of the Supermicro splash screen, it will indicate the BMC IP to access IPMI.
Power Consumption / Heat
On idle, this setup will consume as low as 90W. A Fully load 7-validators setup will run around 250W while a single active THORChain node running on an EPYC 7313P will stay around 140–160W on average, and almost never go above 200W at the most. This should not do a major increase on the electricity bill, and also should not warm up the place. You could leave it in the corner of the living room without even noticing it.
OS Installation
After assembling the hardware together, I downloaded Ubuntu Server 22.04.1 LTS from the official website, created a bootable USB installation stick using the tool etcher, and proceeded to install on the dedicated 480GB SSD.
OS Optimisation (optional)
Disable CPU Mitigation
Disable Swappiness
fsnotify watcher
NVMe Volume
This is an area I improved from previous guides that is worth mentioning. In Hildisvíni Óttar’s guide, we create a simple LMV linear volume; while it does the job and work well, especially on a Vultr server with miss-matched NVMe, it doesn’t stripe data across all devices. That approach will only write to one device at the time, and only start writing to the second device once the first one is full. Striping data across all devices in a RAID-0-like configuration provides a performance increase of having all devices working together, and also distributes usage uniformly across them.
Two available options:
Create a Striped Logical Volume using LVM2:
Very similar to Hildisvíni Óttar’s guide, a logical partition is created using the lvcreate command, but additional parameters are added to specify to stripe the data across all devices.
sudo lvcreate -l {value} -i 4 -I 128k nvmevg0 -n nvmelv0- -i 4 — Number of Stripes (Disk need to be used for stripes)
- -I 128k — Size of a single stripe.
More information on lvcreate parameters can be found here.
Creating a RAID-0 using mdadm:
This is really what I would call a true software RAID. Example below with 12 devices.
sudo lsblk
sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=12 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 /dev/nvme10n1 /dev/nvme11n1
mdadm -D /dev/md0
sudo mkfs.btrfs /dev/md0
sudo mdadm --detail --scan --verbose | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u
sudo mkdir /data
sudo blkid /dev/md0
(copy the UUID for next step)
sudo nano /etc/fstab
add following line:
UUID=b351ec42-5e10-40b2-bd65-ab7c447022df /data btrfs defaults 0 0
sudo mount -a
df -hMore Information about mdadm can be found here.
Pros/Cons:
The advantage of LMV2 over mdadm is that the LMV2 approach will provide a lot more flexibility, allowing to use NVMe of different sizes together, as well as expand the logical partition to additional NVMe drive later if required. While the mdadm approach is closer to a pure RAID-0 configuration, requiring all devices to be the same size (or using the smallest device as a baseline) and preventing from expanding easily later, but would be more performant. No redundancy is used or required here, if an NVMe fail, the node can simply churn out and resync after replacing the defective device. I opted for the second option given all my devices are the same size, and I should have enough capacity to support additional chains. Worst case, I could still add more NVMe and create a second RAID-0 data partition and assign specific pods to it.
APC UPS Tools — apcupsd
Apcupsd is a software package that provides the user with the ability to manage their APC brand UPS.
sudo apt install apcupsd -y
sudo sed -i 's/DEVICE \/dev\/ttyS0/#DEVICE \/dev\/ttyS0/g' /etc/apcupsd/apcupsd.conf
sudo systemctl restart apcupsd
apcaccessMore information on APC UPS Service apcupsd can be found here.
Kubernetes (K3S) default storage paths
One could configure a THORNode on this hardware following Hildisvíni Óttar’s guide as is.
The only exception worth mentioning is the installation of K3S:
curl -sfL https://get.k3s.io | sh -s - - disable=traefik - data-dir=/data - write-kubeconfig-mode 644This extra parameter will allow the installation of all pods without editing configuration to specify a value for the paths. The storage of each pods will automatically be located under /data which simplify the installation process.
Conclusion
I hope this guide will help give alternative option to enterprise server hardware. Should you have any questions please don’t hesitate to reach out to me on Discord.
See Part 2 about configuring multiple THORChain nodes using MicroK8s on a bare-metal server.
