How to monitor your Persistence nodes using PANIC!

Published in

Simply Staking

6 min readAug 17, 2021

In this guide, we will be going through the steps required to set up monitoring and alerting for your Persistence Network validator infrastructure. The software to be used is Simply VC’s open-source alerting tool, PANIC.

Situations alerted:

Missing Blocks
Validator voting power changes
Validator jailing
P2P peer fluctuations
Out of sync nodes
Unreachable nodes
New Github releases

Notification channels:

Logging — alerts logged to an alerts log
Telegram alerts — alerts delivered to a Telegram chat via a Telegram bot
Automated phone calls — alerts trigger a phone call to grab the node operator’s attention
Email notifications — alerts sent by email using a personal SMTP server

Setting up the Dependencies

Python3

Python3 comes preinstalled by default on most Linux distributions. In this guide the Linux release on this machine is Ubuntu 20.04 and python3 was installed by default with the current version being Python 3.8.5. Therefore if it is not preinstalled with your distribution, it is available as a package.

Check if Python3 is installed by running the following command.

$ python3 –-version

Installing PIP Package Manager, Pipenv Packaging Tool and Redis Database Server

# sudo apt-get install python3-pip redis-server -y
# sudo pip3 install pipenv
# sudo systemctl enable redis-server.service

Opening up the Persistence node’s RPC port for monitoring access

By default, the node’s RPC port is only accessible locally, as can be seen in the following snippet of $HOME/.persistenceCore/config.toml.

[rpc]# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://127.0.0.1:26657"

Should you be installing PANIC on a remote host, set the setting to the following value and restart your Persistence node.

[rpc]# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://0.0.0.0:26657"

Installing PANIC

1. Create a new user to run the alerter as:

# adduser panic_alerter --disabled-login

2. Create a directory that will have the PANIC alerter cloned into it. Since this is now owned by the root user, set it to be owned by the new panic_alerter user, then switch to that user:

# mkdir /opt/panic_alerter
# chown -R panic_alerter:panic_alerter /opt/panic_alerter
# su panic_alerter

4. Now logged in as the panic_alerter user, clone the PANIC Github repository:

$ cd /opt/panic_alerter/ && git clone https://github.com/SimplyVC/panic_cosmos.git && cd panic_cosmos && git checkout master

5. Update the project’s packages and run the alerter setup:

$ pipenv update
$ pipenv run python run_setup.py

Initialising the PANIC alerter setup

From here onward, follow the installation wizard and input the API tokens, node info, etc. to configure the different notification channels and the nodes which are to be monitored.

Follow the installation wizard and as we want to set our alerts up with Telegram we are required to insert the Telegram bot’s API token.

In order to obtain the API token, you must have a Telegram account. Download the Telegram app for Andriod or iPhone on your phone and create an account using your phone number.

Once you have your Telegram account set up you can create the bot by the following steps:

Add @BotFather on Telegram and press start.
In the chat with @BotFather send a /newbot command and set the name using the /setname command.
Take note of the API token, which will look something similar to 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11.
Access the link api.telegram.org/bot<token>/getUpdates, replacing <token> with the bot’s API token. This gives a list of the bot’s activity, including messages sent to the bot.
Once you are on the chat with the bot, click on the start option and it will send the /start command to the bot.
Check the api.telegram.org/bot<token>/getUpdates, the result section should contain at least one message. Take note of the “id” in the “chat” section of this message.
One bot will suffice, however you may repeat the above steps to create more bots.

In step 5, once the connection is successful, a Test Alert message will be delivered to you.

On having completed these steps you should have:

A Telegram account
A Telegram bot
The Telegram bot’s API token
The chat id

With these details we can now carry on with the setup. Enter the Telegram bot’s API token. A subsequent prompt will ask you for the chat id in order to sync up the chat with PANIC.

Set up Telegram commands in order to be able to interact with the bot using certain commands. After accepting to set up the Telegram commands you will be prompted for the bot’s API token.

We will then set up redis in order for status updates to be enabled. This will prompt you for the Redis host IP: (default: localhost). You may leave the answer empty and press enter in order to allow the default to be set, which is localhost.

Onwards to the next step we will press enter and allow the default value 6379 to be set for the host port.

Insert the Redis password, if one is set up. (by default Redis doesn’t have a password)

Onto the last Redis prompt, where you should test your Redis connection configuration.

To produce alerts, the alerter needs something to monitor. Therefore in this case you will be required to provide the IP of your validator node. Once added, it will ask if you want to add another node. This is optional as we will be monitoring one validator node.

Once the setup is complete, a “Setup Complete!” message will display on your prompt and you will now be able to run PANIC.

Starting PANIC as a Service

The service file is created as follows:

# nano /etc/systemd/system/panic_alerter.service

Entering the service definition:

[Unit]
Description=P.A.N.I.C.
After=network.target
StartLimitIntervalSec=0[Service]
Type=simple
Restart=always
User=panic_alerter
TimeoutStopSec=90s
WorkingDirectory=/opt/panic_alerter/panic_cosmos
ExecStart=/usr/local/bin/pipenv run python /opt/panic_alerter/panic_cosmos/run_alerter.py[Install]
WantedBy=multi-user.target

Running the alerter as an enabled service means that it starts up automatically on boot and restarts automatically if it runs into some issue and stops running.

You may omit the enabled service command if you do not want the alerter to start up automatically.

# sudo systemctl enable panic_alerter.service

Lastly, we will start the alerter service:

# sudo systemctl start panic_alerter.service

Check out systemctl status panic_alerter.service or the logs in /opt/panic_alerter/panic/logs to confirm that the alerter is running.

Try interacting with the Telegram bot now using the /ping command. To confirm that the bot is responding, it will return “PONG!” as a reply.

Moreover, in order to check the status of your node, you may type /status.

Use /help to find out more ways of interacting with the Telegram bot.

Changing configuration

Configuration options are stored in .ini files in the config directory of panic. Here you may find:

user_config_main.ini for the notification channel setup
user_config_nodes.ini for nodes to be monitored
user_config_repos.in for GitHub repos to be monitored (only if this feature is enabled)

Stay tuned to Simply VC for more PANIC related guides and documentation:

How to monitor your Persistence nodes using PANIC!

Installing PANIC

Starting PANIC as a Service

Changing configuration

Written by Gianluca Cremona