PANIC — Installing an Emergency Notification System for your Cosmos Validator

In this short intro, we’ll go through the steps required to set up push notifications, emails and automated phone calls related to validator uptime, voting power and network statistics. The software to be used is Simply VC’s self-hosted and open-source alerting tool, PANIC.

PANIC ground control

Situations alerted:

• Missing blocks
• Validator voting power changes
• Validator jailing
• P2P peer fluctuations
• Out of sync nodes
• Unreachable nodes
• New Github releases

Notification channels:

• Logging — alerts logged to an alerts log
• Telegram alerts — alerts delivered to a Telegram chat via a Telegram bot
• Automated phone calls — alerts trigger a phone call to grab the node operator’s attention
• Email notifications — alerts sent by email using a personal SMTP server

The source documentation for this article can be found here: Installing and Running P.A.N.I.C.

We’ve previously published an overview of the monitoring options for Cosmos node operators: Monitoring and Alerting for Validators.

Requirements

We’ll be going over the installation steps for the following requirements below:

• Ubuntu 18.04 system
• Python3
• PIP Package Manager
• Pipenv Packaging Tool
• Optional: Redis Database Server
• Optional: Telegram bot
• Optional: Twilio account

Setting up the dependencies

• Python3

Ubuntu 18.04 comes with Python 3.6 by default. You should be able to check Check if Python 3 is installed by running

$ python3 --version

• Installing PIP Package Manager, Pipenv Packaging Tool and Redis Database Server

# sudo apt-get install python3-pip redis-server -y
# sudo pip3 install pipenv
# sudo systemctl enable redis-server.service

• Configuring Redis

1. Access the redis.conf file:

# nano /etc/redis/redis.conf

In the GENERAL section, add the line bind 127.0.0.1 ::1 to bind Redis only to localhost, if it is not yet already added. This assumes that Redis is installed on the same system that will run the alerter.

In the SECURITY section, add (or uncomment) the line requirepass <PASS>, where <PASS> is a complex password of your choosing. The alerter will ask for this password during setup.

requirepass password123

In the SECURITY section, disable some unused commands to prevent some actions from taking place. For each command, add a line: rename-command <COMMAND> "".

rename-command FLUSHALL ""
rename-command PEXPIRE ""
rename-command CONFIG ""
rename-command SHUTDOWN ""
rename-command BGREWRITEAOF ""
rename-command BGSAVE ""
rename-command SAVE ""
rename-command SPOP ""
rename-command SREM ""
rename-command RENAME ""
rename-command DEBUG ""

2. Restart Redis to apply changes:

# systemctl restart redis-server

• Creating a Telegram Bot

Add BotFather on Telegram, press Start and follow the below steps:

1. Send a /newbot command and fill in the requested details, including a bot name and username.

2. Take a note of the API token, which looks something like 123456:ABC-DEF1234ghIkl-zyx56789g1u163eh41

3. Access the link t.me/<username> to your new bot given by BotFather and press Start.

4. Send a /start message to the bot and then any second message.

5. Access the link api.telegram.org/bot<token>/getUpdates, replacing <token> with the bot's API token. This gives a list of the bot's activity, including messages sent to the bot.

6. The result section should contain at least one message, due to us sending a message. Take a note of the "id" number in the "chat" section of this message.

Note: A group chat can also be retrieved by adding the bot to the chat and finding this chat’s ID through the same Telegram API method. Group chat IDs start with a negative sign (-).

• Setting up a Twilio account

1. To create a free trial Twilio account, head to the try-twilio page and sign up using your details.
2. Visit your account dashboard.
3. Click the ‘Get a Trial Number’ button to generate a unique number.
4. Take note of the (i) Twilio phone number.
5. Take note of the (ii) account SID and (iii) auth token.
6. Navigate to the Verified Called IDs page.
7. Press the red + to add a new personal number and follow the verification steps.

Installing PANIC

1. Create a new user to run the alerter as:

# adduser panic_alerter

2. Create a directory that will have the PANIC alerter cloned into it:

# mkdir /opt/panic_alerter

3. Since this is now owned by the root user, set it to be owned by the new panic_alerter user, then switch to that user:

# chown -R panic_alerter:panic_alerter /opt/panic_alerter
# su panic_alerter

4. Now logged in as the panic_alerter user, clone the PANIC Github repository:

$ cd /opt/panic_alerter/ && git clone https://github.com/SimplyVC/panic_cosmos.git && cd panic && git checkout v1.1.2

5. Update the project’s packages and run the alerter setup:

$ pipenv update
$ pipenv run python run_setup.py

Initialising the PANIC alerter setup

From here onward, follow the installation wizard and input the API tokens, node info, etc. to configure the different notification channels and the nodes which are to be monitored.

Starting PANIC as a Service

Running the alerter as an enabled service means that it starts up automatically on boot and restarts automatically if it runs into some issue and stops running.

The service file is created as follows:

# nano /etc/systemd/system/panic_alerter.service

Entering the service definition:

[Unit]
Description=P.A.N.I.C.
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
User=panic_alerter
TimeoutStopSec=90s
WorkingDirectory=/opt/panic_alerter/panic
ExecStart=/usr/local/bin/pipenv run python /opt/panic_alerter/panic/run_alerter.py

[Install]
WantedBy=multi-user.target

Lastly, we will enable and start the alerter service:

# sudo systemctl enable panic_alerter.service
# sudo systemctl start panic_alerter.service

Check out systemctl status panic_alerter.service or the logs in /opt/panic_alerter/panic/logs to confirm that the alerter is running. Alternatively, if you set up Telegram, try interacting with the Telegram bot (using /help).

Changing configuration

Configuration options are stored in .ini files in the config directory of panic. Here you may find:

• user_config_main.ini for the notification channel setup
• user_config_nodes.ini for nodes to be monitored
• user_config_repos.in for GitHub repos to be monitored (only if this feature is enabled)

Part 2 of this guide will outline the following:

• Using PANIC
• Node and Network Monitors
• Alert levels and customisation
• Interacting with the Alerter through Telegram
• Operational recommendations

Stay tuned to Simply VC for more Cosmos related guides and documentation:

• Website: https://simply-vc.com.mt
• Twitter: https://twitter.com/simply_vc
• Telegram: https://t.me/SimplyVC

Feel free to contact me with any questions or feedback here:

• Telegram: https://t.me/FrancescoSimplyVC
• Twitter: https://twitter.com/cremonafran