Flux Multi-Node Server Setup

Setup a dedicated server to run multiple instances of Flux node, with only 1 public IP address (Hetzner Specific)

In this article, I will guide you through the process of setting up a dedicated server to run multiple flux node instances, each running within a VM (virtual machine), setup to use UPnP (Universal Plug and Play) to operate with just a single public IP address.

If you are not planning to run multiple nodes on a single system, this is the wrong guide for you. Follow Step 1 (Installing Ubuntu instead) and then follow this guide instead: Flux Light Node Setup — As easy as it gets !! | by Ali Malik | Medium

Prerequisites

1. The required Flux collateral per node you wish to host

For every node you host, you will need to lock up the required Flux collateral for that node’s tier per node. For more information, please see: Flux Light Node Setup — As easy as it gets !! | by Ali Malik | Medium

2. A dedicated server to run your node instances

In this guide, I will walk you through the setup of an AX101 from Hetzner. This server can handle up to 2 Stratus nodes, or a combination of all 3 tiers (to a maximum of 8 nodes, as per Flux’s maximum per public IP). See here for details on all possible configurations: Copy of FLUX Node Profitability — Google Sheets

You are free to use whatever hardware and provider you like, however you may find some steps are different (especially the initial setup). When in doubt, always refer to the provider’s documentation

Alternative providers: FluxNodesRentingOptions — Google Sheets

Note: When you purchase your server, make sure to get it without an OS. We will be reinstalling over the top of any OS the server comes with anyways.

3. One additional IP address

You will require a 2nd public IP address from your provider. Hetzner offer additional IP addresses at $2 a month per extra address. This can be purchased after you have access to your server. This is required.

Lets get started

Step 1 - Initial setup and OS install

When you first get your server, you wont have an OS (instead your host machine will have a “rescue-system”). For our purposes, we will need to install Proxmox with RAID 0.

Step 1.1 - Create an SSH connection to your server

For this process, you’ll need an SSH application. You can either use Windows CMD, or an SSH application such as PuTTY. The choice is yours. In this guide, I’ll be using Windows Terminal, which you can get from the Microsoft app store here: Get Windows Terminal — Microsoft Store en-GB

Open your SSH application, and create a connection with the IP address supplied to you from Hetzner (this will be sent as an email). The username will be root, IP address that which is given in the email, and password will also be in the email

Step 1.2 - installimage

Run the command installimage

A GUI will appear with a list of operating systems. For our purposes, we will be installing Proxmox

Use the arrow keys to traverse to Other (!!NO SUPPORT!!)

Press enter and select Proxmox-Virtualization-Enviroment-on-Debian-Bullseye

Don’t worry about the warnings about the image not being supported by Hetzner.

On the next screen, you’ll be presented a config file that you can edit to change parameters of the installation. Make the following changes:

1. Change SWRAIDLEVELto 0
2. Under Partitions/Filesystems, after all the #’s, remove the line PART /home ext4 all
3. Change PART / ext4 2014Gto PART / ext4 all

Once you are done, press F10 and select Yes

Ignore the following warnings, they are just telling you the drives will be wiped.

Let the installation run. This can take up to 10 minutes.

Once the installation is complete, enter the command rebootto restart the server and boot into Proxmox. The server can take 5 minutes or more to restart. You will not be able to establish an SSH connection during this time

To check Proxmox has installed correctly, re-establish an SSH connection with your server, using the same credentials. root@Proxmox~# should be shown in the terminal window.

Note: If you get an error warning you that Remote Host Identification has change, you’ll need to run the command ssh-keygen -R <host> in the command prompt, where <host> is the IP address of your server

To access your Proxmox web interface, go to https://<yourIP>:8006 where <yourIP> is the public IP address you used to SSH to your server

Step 2 - Proxmox network configuration

Now we have Proxmox installed, we need to setup the network. We will create 2 interfaces within Proxmox and edit the /interfaces file

Step 2.1 - Record your MAC address

Create an SSH connection to your server, ensuring you are root within Proxmox

Run the command ip -o link and note down the MAC address of the network interface of your server. This will be the address that looks like XX:XX:XX:XX:XX:XX with numbers and letters, at the end of the of the line with an interface called something similar to “enp7s0”

Step 2.2 - Edit the interfaces file

Note: Be very careful in this step to ensure all addresses are correctly inputted. Any mistakes will cause your server to become unreachable over the internet. If this happens, you will need to contact Hetzner support to request a KVM connection, where you can either restart from Rescue, or try to fix your mistake in the interfaces file. You have been warned.

Create an SSH connection to your server, ensuring you are root within Proxmox

Enter the command nano /etc/network/interfaces

This will open your interfaces file. Delete everything in this file and paste in the following:

source /etc/network/interfaces.d/*
auto lo 
iface lo inet loopback
  
iface <NIC NAME (WILL BE SOMETHING LIKE "enp7s0")> inet manual  
auto vmbr0 
iface vmbr0 inet static   
address <HOST IP/CIDR (THIS WILL BE PRESENT WHEN YOU FIRST OPEN THE FILE>
gateway <GATEWAY IP (PROVIDED BY HETZNER, MOUSE OVER YOUR SERVER'S IP FOR INFO)>  
bridge-ports <NIC NAME>  
bridge-stp off   
bridge-fd 1   
hwaddress <MAC (PREVIOUSLY RECORDED)>
pointopoint <GATEWAY IP>
auto vmbr100 
iface vmbr100 inet manual   
bridge-ports none   
bridge-stp off   
bridge-fd 0

Change the values enclosed with <> to those of your server.

Once done, press CTRL + X, press y, then press enter.

Now enter reboot to reboot your server

This is how my interfaces file looks:

Ensure that you can still access your server, both through SSH and your internet browser before proceeding

Step 3 - Create the Virtual Router

At this point, we are ready to create the Virtual Router. This VM will control all network traffic to and from your node VMs, and will act as the gateway for your node network. We will be using OPNsense, however feel free to use any other router software such as pfsense

Note: You can choose to download the file locally and then upload it to your server (steps 3.1.a and 3.2.a) or handle everything on the server (step 3.1.b) Whichever method you go for, feel free to skip the other method’s steps

Step 3.1.a - Download the OPNsense Image (locally)

This method will have you download the .iso file to your local device, and then upload to the server. This is easier, however the file is over 1GB and so if you have a slow internet connection, I recommend you skip to Step 3.1.b where we will do the same process straight from the server

Skip to step 3.1.b if you wish to do this step on the server

You will need to download the OPNsense image and upload it to your server first.

Navigate to the OPNsense downloads page here: Download — OPNsense® is a true open source firewall and more

Select amd64, dvd, and a location close to you.

Download this file and extract the .iso file to a directory that is easy to navigate to (right click the .bz2 file and click extract files… then choose a path).

Step 3.2.a - Upload the ISO file (locally)

Now in your Proxmox server, navigate to the ISO images section by clicking on your storage (called local), and then ISO images. From here, click upload.

In the upload window, select your ISO file you just downloaded and click upload

The upload may take sometime, depending on your network speeds. Wait until the upload is complete to proceed.

Step 3.1.b- Download and upload the ISO file (on host)

Skip if you have completed steps 3.1.a and 3.2.b

This process can be done entirely on your server, which is the way I suggest, however it is more technical.

Open an SSH window to your server, and type the command cd /var/lib/vz/template/iso

Now, go to step 3.1.a and follow it to the point where you are asked to click download. Instead of downloading this file, we are going to right click and copy link

Now type wget <PasteYourLinkHere> where you replace with your link (without the <>)

After a minute, the download should be complete.

As you can see, the file is now in the directory, however we need to decompress it.

Run the following commands

1. sudo apt install bzip2 This may do nothing, that is ok
2. bzip2 -d <filename.bz2> Where <filename.bz2> is the name of the file you just downloaded. You can see that in the line above or by typing ls -a

After you’ve ran the second command, nothing will appear to happen for a minute or two. Just wait until the text appears again on the left

To verify this has worked, type ls -a where you should see the .iso file without the .bz2 extension

Step 3.3 - Configure the VM

In the top right corner, click Create VM

Give the VM and ID of 999 and a suitable name. Click Next

In OS, select the OPNsense ISO file we’ve just uploaded and click next.

Leave everything as default in the System tab

In the Disks tab, change the disk size to 8 (this is 8GB) and click next

In the CPU tab, give the VM 2 sockets per core.

In the Memory tab, change the Memory to 2048 and click next

Now, if you’re using Hetzner, go to your additional IP address (if you haven’t ordered one yet, this is the time) and either click request separate MAC or, if you have already done this, mouse over your additional IP address and copy the MAC address.

Ensure that Bridge is set to vmbr0 and then paste your MAC address into the MAC address field. Click next

Ensure everything is set correctly and click finish

Step 3.4 - Add a second network card

Now that you have created your VM, you will see it on the left with the ID and name you gave it. We must now add a second network card, which will be our vmbr100, which will connect the LAN that will hold our VMs in.

Click on your router VM, then click Hardware, then add, and select Network Device

Set the Bridge to vmbr100 and click add

You have successfully added your Virtual Router VM

Step 4 - Configure OPNsense

Now that we have the network setup, and a VM running OPNsense with the correct network interfaces, we now need to setup OPNsense to allow us to run multiple nodes in the same LAN

Step 4.1 - OPNsense installer

In Proxmox, click on your router VM and press start in the top right corner.

Now go to the Console tab of your VM, and wait until the text appears on screen

After the boot has finished, you will be prompted to login. Type installer and press enter

For the password, type opnsense

In the next screen, select the keyboard layout you wish to use

On the next screen, leave the setting on default (GPT/UEFI Hybrid) and press enter

In UFS Configuration, select da0, which should have 8GB of space. Press enter

Select yes to confirm the warning on the next page.

Once the installation is complete, you can choose to change the root password and then select Complete install.

The machine will now reboot

Step 4.2 - OPNsense WAN interface

Boot the VM, and open the Console tab to access the VM.

If you have left the credentials as default, login with the username root and password opnsense

For now, WAN need to be assigned to vtnet0, and LAN needs to be unassigned (we will set the LAN later)

Select option 1.

We do not wish to configure LAGGs or VLANs, so press N and enter for both

For our WAN interface, enter vtnet0

For our LAN interface, skip by pressing enter

Skip the optional interface by pressing enter again.

If the assignment looks correct, enter y

The router will now configure the interfaces. Once this has finished, ensure your interfaces are now correct

We will be assigning the LAN later

Step 4.3 - Ports

To access the web gui interface of your OPNsense router, type https://youripaddress where your ip address is that of the router (found in the previous step as the WAN IP address

Login with your credentials (root and opnsense by default)

Once logged in, click on Firewall, NAT, Port Forward.

On the right, click add another record.

In this form, change the following fields:

1. Destination = WAN address
2. Destination port range = from (other) 80 to (other) 443
3. Redirect target IP = 192.168.1.1

Click save and then apply changes on the next page.

Your port forwarding tables should now look like this:

Step 4.4 - Enable SSH (OPTIONAL)

You can add a third rule to forward SSH connections (port 22) in the same way as we did in the previous step. This will allow you to SSH into your router, which you can then SSH hop to your nodes.

Configure a third rule like you see below

Next, navigate to administration, which is located in the settings tab of System, and do the following changes:

1. Enable Secure Shell
2. Permit root user login
3. Permit password login

Click save

You can now access your router over an SSH connection. To test this, open an SSH window, now using your router’s IP address and login credentials.

To access the shell, select option 8

Step 4.5 - LAN interface

You can now hop back over to your console in Promox for your router

Once logged in, run option 1 again

Like before, say no to both LAGG and VLAN

Set WAN to vtnet0 like before

Set LAN to vtnet1

Again skip the optional interface

Ensure your interfaces are correct and then allow the configuration to proceed.

Your LAN and WAN interfaces should now be setup correctly

Reboot the router

Step 4.6 - Enable UPnP on the router

The next step is to setup UPnP. This will allow us to host multiple nodes (up to 8) on a single Public IP address

In the OPNsense web gui, navigate to the status section in firmware and click check for updates

After a couple seconds, a pop-up will appear. Close this pop-up and then, at the bottom of the screen, click update

Click ok to the firewall pop-up

Wait for the updates to complete and device to reboot

Once the router has rebooted, under Plugins, search for upnp and click install

Let the install complete and then refresh your browser page.

Next, navigate to Universal Plug and Play settings, under Services and make the following changes:

Note: For number 8, you cannot copy and paste Entry 1 as the “-” used by the router is a different character from the “-” used in medium’s blog

1. Enable = ticked
2. Allow UPnP Port Mapping = ticked
3. Allow NAT-PMP Port Mapping = ticked
4. External interface = WAN
5. Interfaces (generally LAN) = LAN
6. Log NAT-PMP = ticked
7. Use system time = ticked
8. Entry 1 = allow 1–65000 192.168.1.1/24 1–65000

Ensure you’ve configured everything correctly and click save

Step 4.7 - Disable DHCP

Although this step is not entirely necessary, it will make things massively easier later on

Navigate to [LAN] in DHCPv4, within services

Ensure the setting Enable is not ticked

Click save at the bottom of the page

Step 5 - Create VMs for your nodes

At this stage, you are ready to start creating VMs that will host your nodes. Each node needs its own VM, and depending on the node tier you wish the VM to run will change what hardware configurations the VM will need

For purposes of this guide, I will be setting up cumulus nodes.

Step 5.1 - Download and install Ubuntu 20.04

Much like what we had to do for OPNsense, we need to download and upload the Ubuntu .iso to our server to give to our VMs. This can be done much more simply than in previous steps, however feel free to complete this step however you see fit

Navigate to Ubuntu’s download page here: Get Ubuntu Server | Download | Ubuntu

Click Download Ubuntu Server

This will start a download and take you to another page. You can cancel this download and instead right click and copy link

Now in Proxmox, navigate to ISO Images, which is in your local storage, and click download from URL

Paste your link and click Query URL. This should automatically pull the .iso file we need. Now click download

After the download is complete, we have the image we need to install Ubuntu onto our VMs

Step 5.2 - Create the first VM

Click create VM and go through the tabs, configuring the VM like you did for the Router, but instead using the hardware requirements for your node of choice

Note: It is better to have some overhead in your hardware. I suggest you give each VM 20GB extra than is required, and feel free to give a little extra RAM if you can spare it.

For OS, select our Ubuntu ISO image we just downloaded

Leave System as default

Disks will depend on your node tier. For my, I wish to setup a Cumulus node, and so will give the VM 240GB (220GB minimum + 20GB overhead)

In CPU, give your VM their required cores and threads. For a Cumulus, this would be 2 cores with 2 threads.

For RAM, give your server the required RAM for the tier. For a Cumulus, this would be 8GB (which is 8192MiB)

Note: 1GB = 1024MiB

In Network, ensure to set all your VMs to Bridge vmbr100

Click next then Finish.

Step 5.3 - Install Ubuntu OS

Navigate to the console tab of your newly created VM and start the machine

After initialization, you will be presented with a list of languages. If this is not the case, you have done something wrong.

Select your preferred language and press enter

On the next page, select your preferred keyboard layout and select done

For IPv4, Select edit, then manual.

Now you will need to configure the IP addressing of your VM. Set each setting to the following:

1. Subnet = 192.168.1.0/24
2. Address = 192.168.1.X where X is between 2 to 254 inclusive, and unique to this VM
3. Gateway = 192.168.1.1
4. Name Servers = 8.8.8.8
5. Search Domains = 8.8.4.4

Select Save

Skip past proxy configuration

Skip past Ubuntu Mirror Archive

In guided storage configuration, ensure Use entire disk is selected, and the disk shown has the size you gave your VM. Ensure also that Set up this disk as an LVM group is ticked

On the next page, a few changes need to be made

At the top of the page, select the “/ “ mount point (it will have a size of 100GB) and select Unmount

Next, select Ubuntu-lv under devices (it will have a size of 100GB), and select Edit

Within the edit window, increase the size to the max value (you can just make this number huge and Ubuntu will reduce it to the max size)

Then set the mount to “/ ”. Click Save

Ensure that the “/ “ directory has enough Size to meet the system requirements of your node. Then click Done

Select continue on the confirmation pop-up

Now edit the fields on the next page. These can be whatever you want, but make sure to remember your username and password

Skip the Ubuntu advantage token page

OPTIONAL - If you wish to be able to SSH into your VM (HIGHLY RECOMMENDED), select Instal OpenSSH Server

Note: You’ll have needed to follow the previous optional steps for SSH in order for SSH to work correctly

On the next page, press tab to skip the list of options and select done

Now allow the VM to install Ubuntu. The installation may take up to 10 minutes. Once it has finished, you will be prompted to reboot the machine.

Once you select reboot, a list of errors will appear. This is normal. Press enter to continue

Once your VM has rebooted, use the previously set credentials to login to the machine.

Step 5.4 - Update and Upgrade Ubuntu

Now that Ubuntu is installed onto your VM, you’ll need to update the OS and any files.

Run the command sudo apt update && sudo apt dist-upgrade

You’ll be prompted for your user password, enter it and let the command roll

Once the VM has pulled all the updates, it will ask if you wish to continue. Enter Y

Once the updates are finished, enter the command sudo reboot to reboot your VM

A note on SSH: If you wish to SSH into your nodes, you’ll need to first establish an SSH connection with your router, press option 8, then SSH into your nodes from here. You cannot access the nodes without accessing the router.

The syntax for an SSH connection in CLI is SSH username@address

The username will be whatever you set when installing the OS (or root for root access)

For example:

SSH to Router

SSH to Node

To leave the node and go back to the router, type exit

Step 6 - Install Flux

At this point, you are ready to install flux onto your node. You should follow this awesome guide by Ali Malik on how to install flux: Flux Light Node Setup — As easy as it gets !! | by Ali Malik | Medium

Note: Ensure you are connected to the VM you intend to install Flux onto. For now, just setup 1 machine from start to finish. We can create clones of this VM in the future

Be sure to complete all the steps of this guide, and then return to this guide at the end of that guide to configure UPnP on your flux nodes

Helpful tip: If you need to change from your user account to root, use the command sudo su -

Give zelcore your router’s public IP address for the node’s IP

You will fail benchmarks until you enable UPnP

Step 7 - UPnP and Flux

At this stage, you will have just completed the flux node install, and are ready to enable UPnP on your node

Step 7.1 - Run option 14

In the multitoolbox, run option 14

After a few seconds, a pop window will appear. Select Enable UPnP Mode

Now you will need to select a port to use. Each node will need to run a different port.

If you configured everything correctly, flux should auto detect your router address as 192.168.1.1

Select Yes and allow UPnP to install

Congratulations. You are done. This node will now run on flux.

Step 8 - Create clone-able VMs (OPTIONAL)

At this stage, your first node is now fully operational. This final optional step will allow you to create a template of your VM, so that you may create multiple nodes quickly and have the ability to scale

Note: This process will destroy your initial VM. This is fine, as we will create a clone of the template, which will be virtually identical to the VM before it was made into a template

Note 2: You cannot create a template out of a cumulus node, and then try to change the hardware and deploy a Nimbus node. You must configure the VM with the tier in mind, set it up, and then create a template from it. If you wish to have a template of all 3 tiers, you must initially setup 3 separately speced VMs

Step 8.1 - Create a template

In Proxmox, right click your running VM and click Shutdown

Your node will now shutdown. Next, right click and click Convert to template

This will irrevocably covert your VM into a template.

Step 8.2 - Clone your Node

To create a VM from this template, right click and click clone

Change mode to Full clone and give the VM a suitable name. Click Clone

Your cloned machine will now appear in the list powered off. Power on the machine and either connect through the console, or establish an SSH connection to the VM (you can find the IP address of your cloned device by first logging into it from the Proxmox web Console)

Step 8.3 - Change IP address

Each VM needs a unique IP address. To change the IP address of your VM, enter the following command:

sudo nano /etc/netplan/00-installer-config.yaml

This will open a file, much like the /interfaces file we edited earlier. Here, it should already be pre-populated with information. All we are going to change, is the addresses from 192.168.1.100 to something else

Once you have changed the IP address, hit CTRL + X, y, then enter

Now type sudo netplan apply

Note: If you are connected to your machines via SSH, your SSH connection will now become unresponsive until it times out. This is because the target IP address of your machine has changed

Step 8.4 - Change Flux Config

As your node is a clone of a machine that already has Flux fully installed, there is no need to re-install flux. Periodically, you may wish to refresh your templates as flux updates

Open the Flux Multitoolbox and select option 8

bash -i <(curl -s https://raw.githubusercontent.com/RunOnFlux/fluxnode-multitool/master/multitoolbox.sh)

Here, you will amend your node’s details for the node you are trying to setup in zelcore

After this has ran, re-call the toolbox and run option 14

Enable UPnP mode

It is essential we choose a port that is not currently in use by any other of your nodes. I will select 16127 for this particular machine

Make sure your router’s IP address is 192.168.1.1

Let this finish running, and then call the multibox a final time.

Now call option 4. We are reinstalling watchdog to ensure it is properly functioning, and to have this new node give notifications for itself.

Configure Watchdog however you wish

Conclusion

Congratulations for making it this far, you’ve completed the installation.

Learn more about flux at Flux — Decentralized Cloud Infrastructure (runonflux.io)

If you are having any issues, feel free to hop into the Flux Offical discord and post a question in the support channel https://discord.io/runonflux

Or ping me directly on Discord @Goldie ¯\_(ツ)_/¯#0001

Note: This guide is not official or from the Flux team, I am just a community member who wants to help :)

This guide took a long time to write. If you found it helpful, please consider chucking me a flux or 2 ❤️

t1fj6hgzxSEJEqwgWCyF8TJ57zRwffEneGF