WAX Technical How To #11
Running WAX software can be very demanding on compute resources especially on the extremely busy WAX Mainnet. Other than the requirement for a snappy CPU, WAX Software runs the entire network in resident memory and maps this memory to a thin provisioned memory mapped file called the state database which means that some lofty RAM and Disk IO requirements also need to be met.
This 11th WAX Technical How To Edition will work through an example of managing RAM and Disk requirements in an effectively and economical way.
Optimising RAM and Disk Utilisation
There are two challenges this article will address, managing usage of RAM and managing the amount of Disk IO. RAM is costly and often limited on a specific hardware platform and Solid State Disks have a limited lifetime based on the amount of writes that occur (SSD Endurance).
At time of writing the WAX Mainnet’s memory utilisation is 81.4GB and growing by the minute.
Currently there are two effective mechanisms to manage RAM and Disk IO. The first is using WAX Software’s
heap mode, which preloads the state database into swappable memory on node start. The second and what this article’s example will walk through is utilising the Linux Kernel’s tmpfs mechanism and a large amount of disk swap.
tmpfs is a Linux file system that keeps all of it’s files in virtual memory, the contents of this folder are temporary meaning if this folder is unmounted or the server rebooted all contents will be lost.
It has been discovered through testing by the community (Aaron Cox and Stan cc32d9) that much of the resident memory data isn’t accessed very frequently and can be swapped to disk. This understanding means we can be far more efficient on our RAM utilisation and Disk IO demands.
Summary of the tmpfs memory strategy
This strategy is currently working for EOSphere in our production environment, even on nodes that have enough RAM. Offsetting the constant writes to disk appear to be saving our SSD’s. At time of writing a production node is able to run effectively configured like this with only 40GB of RAM.
In summary this example will do the following:
- Mount a tmpfs folder
- Create a large swap file
- Configure nodeos to use the tmpfs folder for state
- Start nodeos from a snapshot
This example will have a 128GB State Database which will be placed in the tmpfs folder which will be 129GB to ensure it‘s big enough’.
fstab as below:
> sudo nano /etc/fstabtmpfs /home/eosphere/waxdata/state tmpfs rw,nodev,nosuid,size=129G,x-systemd.after=zfs-mount.service 0> sudo mount -a**check that is was mounted**> df -h
Linux swap can be configured in it’s own partition or as a file, from our experience using SSD’s (non spindle disks) there hasn’t been any issue using a file and gives us the ability to easily adjust the swap size.
Swap will be configured as 128GB as below:
***Turn off existing swap, which would probably be a partition***> sudo swapoff -a> sudo fallocate -l 128G /swap.img> sudo chmod 600 /swap.img> sudo mkswap /swap.img> sudo swapon /swap.img***Configure fstab and comment out the old swap statement***> sudo nano /etc/fstab/swap.img none swap sw 0 0
nodeos and snapshot start
Ensure that the state database is at least 128GB for now, this will need to monitored as network available memory is constantly expanding.
Configure as below:
***Configure State Database Size***
> cd ~/waxdata
> nano config.inichain-state-db-size-mb = 131072
Start as below:
nodeos --data-dir ~/waxdata --config-dir ~/waxdata --snapshot ~/waxdata/snapshots/snapshot.bin
Start up will take longer than usual as a new state file is created from the snapshot as well as even longer if the nodes physical RAM is exhausted and swap is used. In the case of 40GB of RAM this took about 30mins.
Subsequent restarts of nodeos won’t require a snapshot to start unless your node is rebooted or the tmpfs unmounted.
Of course as already mentioned the WAX Mainnet memory requirements are constantly expanding, regular monitoring of usage and performance will need to guide your hardware requirements and configuration.
Be sure to ask any questions in the EOSphere Telegram
EOSphere Guild is a Block Producer on the WAX Protocol Network as well as many other EOSIO based Blockchains.
If you find our work helpful, please vote for us on the WAX Mainnet: eosphereiobp
If you prefer to proxy your vote, our proxy account is : blklotusprxy
Connect with EOSphere via these channels: