How to keep Denali node alive on Linux

AVA team has released the final version of their ava node before the main net launch this summer. One of the requirements to participate and be rewarded in the incentivized test net is to maintain at least 50% percent up time which equals to 180 hours of up time for the next two weeks.

A healthy, robust network consists of highly operational nodes that are resistant to power shortages, system failures and software crashes. While we can not interfere with the first two requirements, there are a few options to take with regard to the third option.

Let’s start from the easiest but least robust solution and work our way through more resistant alternatives.

Using ‘&’

Ampersand starts the process in the background and makes the current shell available to the user.

./path/to/workspace/gecko-0.5.0/ava &

When parent process of the ava binary (which is the terminal that is being used) is terminated, it sends a SIGHUP signal to its child processes. This signal ends the child process, unless your shell is programmed to do otherwise. With default shell configuration ava node is alive as long as our terminal is up.

You can check if terminal is sending SIGHUP on exit with the command below:

shopt | grep hupon

if the output is huponexit off , ava won’t be terminated when the shell exits.

ava process can be killed anytime, with the command below :

kill -9 `pgrep ava`

Using nohup

Nohup is the stronger version of the ampersand. Because it catches the SIGHUP and ignores it while ampersand does not.

Note : As I was getting “NAT traversal has failed. If this node becomes a staker, it may lose its reward due to being unreachable.” error right after running the binary. I was advised on the official AVA Discord channel to use — public-ip x.x.x.x option. That indeed stopped the warning, so I will be using it in the coming examples.

ava node can be run with nohup as following. Note that we redirect the stdout of the program to an arbitrary file called ava.log located in the directory where ava is.

nohup ./ava --public-ip `curl https://ipinfo.io/ip` > ava.log &

You can see the stdout of the ava node by observing the ava.log file as it is updated constantly.

You can kill the process, as described above :

kill -9 `pgrep ava`

Using systemd

As I have described how linux services are useful in my previous blog post, I will spare you the details and get right into the business. Needless to say this method persists after reboots, shutdowns, program crashes as it will be automatically run again.

Let’s create the service file with the touch command :

sudo touch /etc/systemd/system/denali.service

Then fill it with the following lines using any text editor.

[Unit]
Description=Denali test node
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=YOUR_USER_NAME
SyslogIdentifier=denali.service
ExecStart=/bin/sh -c 'exec /YOUR/AVA/PATH/gecko-0.5.0/ava --public-ip "$(/usr/bin/curl https://ipinfo.io/ip)"'
[Install]
WantedBy=multi-user.target
[Install]
WantedBy=multi-user.target

Note that you have to change User field with your linux user . You can see which user you are logged as with whoami command.

And YOUR/AVA/PATH/ with the absolute path of your gecko-x.y.z folder.

Save and exit the file. Then start your service and enable it so that it persists after reboot.

sudo systemctl start denali.service
sudo systemctl enable denali.service

It will prompt you for your password for authentication. Enter it and you are done .

If you wish to stop the service from bringing the node up constantly , do the following :

sudo systemctl stop denali.service
sudo systemctl disable denali.service

Testing

Let’s test if the service is actually up.

sudo systemctl | grep denali

the output should look like this :

Check if ava node is up with :

ps -aux | grep ava

You should see ava being highligted on the screen.

kill the process either with

killall ava 

or

kill -9 `pgrep ava`

and observe it coming back online with a different PID using the ps -aux command.

You can check the logs your node is printing with

sudo journalctl -f command. Check the logs which are labelled as denali.service. journalctl will keep updating until you kill it with ctrl + c.

Warning

Currently by the time service starts running , network stack is still down. For that matter /usr/bin/curl https://ipinfo.io/ip return empty. I tried to make denali service depend on systemd-networkd-wait-online.service so that it would wait for the network to come online but dependency service didnt work properly on my machine and ended up giving up. You can either kill the ava node once after reset so that it starts with IP or leave it as it is since I am not even sure if — public-ip argument is needed.