Monitoring OKP4 validator node with Tenderduty

Tenderduty is a comprehensive monitoring tool for Tendermint networks. Its main function is to alert a validator if it is missing blocks, and it has many other features.

V2 adds a web dashboard, a prometheus exporter, telegram and discord notifications, multi-chain support, more granular alerts and more alert types.

More info:

GitHub - blockpane/tenderduty: Notification tool for Cosmos/Tendermint validators, sends alerts…

Table of contents

· Table of contents
· What is it used for?
· Install Tenderduty
∘ Update packages and Install dependencies
· Install go
· Add user and install tenderduty
· Application settings
· Service with SystemD
∘ Create Service file
∘ Enable & start service
∘ Check logs
∘ Restart service
∘ Launch Tenderduty
· Set up a Telegram alert
∘ Create a bot
∘ Group alarm
∘ Telegram settings

What is it used for?

Monitors a validator’s performance.

The main purpose of tenderduty is to monitor the status of the consensus and alert if the validator is losing blocks. This can be based on consecutive failures or on a percentage of failures within the cutoff window.

• Alert if jailed, tombstoned or inactive.
• Alert destinations can be customised for each chain.
• Monitors the health of nodes:
• Provides a prometheus exporter for integration with other display systems.
• Sends an alert to any of three destinations: Pagerduty, Discord or Telegram.

Dashboard to visualise the status.

• Displays a table with the status of the validator and the node.
• The last 512 blocks are displayed in the status grid.
• Optionally displays a real-time log message stream with details of ongoing status checks.

Install Tenderduty

Update packages and Install dependencies

sudo apt update && sudo apt upgrade -y 
sudo apt install curl build-essential git wget jq make gcc tmux pkg-config libssl-dev libleveldb-dev tar -y

Install go

ver="1.18.2" 
cd $HOME 
wget "https://golang.org/dl/go$ver.linux-amd64.tar.gz" 
sudo rm -rf /usr/local/go 
sudo tar -C /usr/local -xzf "go$ver.linux-amd64.tar.gz" 
rm "go$ver.linux-amd64.tar.gz" 
echo "export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin" >> ~/.bash_profile source ~/.bash_profile 
go version

Add user and install tenderduty

sudo useradd -rs /bin/false tenderduty 
cd $HOME 
git clone https://github.com/blockpane/tenderduty 
cd tenderduty 
go install 
cp example-config.yml config.yml

Almost everything in tenderduty is controlled via the config.yml file

Application settings

Open config.yml file:

sudo vi $HOME/tenderduty/config.yml

It is necessary to indicate some values to configure the monitoring of a node, such as a name to identify the node, the chain-id, and the valoper address.

# The user-friendly name that will be used for labels. Highly suggest wrapping in quotes.
  "OKP4":
    # chain_id is validated for a match when connecting to an RPC endpoint, also used as a label in several places.
    chain_id: okp4-nemeton-1
    # Hooray, in v2 we derive the valcons from abci queries so you don't have to jump through hoops to figure out how
    # to convert ed25519 keys to the appropriate bech32 address
    valoper_address: okp4valoper1vul4qp50x02j9qc6wcs4zt9f2y22yrl7u0ukyh
    # Should the monitor revert to using public API endpoints if all supplied RCP nodes fail?
    # This isn't always reliable, not all public nodes have websocket proxying setup correctly.
    public_fallback: no

Alerting on consecutive missed blocks the config for a chain could be:

# Controls various alert settings for each chain.
    alerts:
      # If the chain stops seeing new blocks, should an alert be sent?
      stalled_enabled: yes
      # How long a halted chain takes in minutes to generate an alarm
      stalled_minutes: 5
 
      # Most basic alarm, you just missed x blocks ... would you like to know?
      consecutive_enabled: yes
      # How many missed blocks should trigger a notification?
      consecutive_missed: 2
      # NOT USED: future hint for pagerduty's routing
      consecutive_priority: critical

In the next section we introduce an RPC Endpoint.

# This section covers our RPC providers. No LCD (aka REST) endpoints are used, only TM's RPC endpoints
    # Multiple hosts are encouraged, and will be tried sequentially until a working endpoint is discovered.
    nodes:
      # URL for the endpoint. Must include protocol://hostname:port
      - url:  https://okp4-testnet.rpc.kjnodes.com:443
        # Should we send an alert if this host isn't responding?
        alert_if_down: yes
      # repeat hosts for monitoring redundancy
      - url: https://okp4-testnet-rpc.polkachu.com
        alert_if_down: no

You can search for more RPC Endpoint among the validator community resources here:

OKP4 Community of Validators Resources

In case you are running Tenderduty on the same server as the OKP4 node you can use the node’s own endpoint as follows:

# This section covers our RPC providers. No LCD (aka REST) endpoints are used, only TM's RPC endpoints
    # Multiple hosts are encouraged, and will be tried sequentially until a working endpoint is discovered.
    nodes:
      # URL for the endpoint. Must include protocol://hostname:port
      - url: tcp://localhost:26659

Service with SystemD

Create Service file

sudo tee /etc/systemd/system/tenderdutyd.service << EOF 
[Unit] 
Description=Tenderduty 
After=network.target 
 
[Service] 
Type=simple 
Restart=always 
RestartSec=5 
TimeoutSec=180 
 
User=tenderduty 
WorkingDirectory=$HOME/tenderduty 
ExecStart=$(which tenderduty) 
 
# there may be a large number of network connections if a lot of chains 
LimitNOFILE=infinity 
 
# extra process isolation 
NoNewPrivileges=true 
ProtectSystem=strict 
RestrictSUIDSGID=true 
LockPersonality=true 
PrivateUsers=true 
PrivateDevices=true 
PrivateTmp=true 
 
[Install] 
WantedBy=multi-user.target 
EOF

Enable & start service

sudo systemctl daemon-reload 
sudo systemctl enable tenderdutyd 
sudo systemctl start tenderdutyd.

Check logs

sudo journalctl -fu tenderdutyd

Restart service

With any change in the configuration you must restart tenderduty:

sudo systemctl restart tenderdutyd && sudo journalctl -fu tenderdutyd

Launch Tenderduty

Once we start the service you will be able to see the logs:

If everything has worked correctly we will be able to see our dashboard:

http://<IP server>:8888

Set up a Telegram alert

Create a bot

Open Telegram and search for the bot @BotFather:

To create a new bot enter the command “/newbot”.

Follow instructions he given and create a new name to your boot:

Once we have completed the steps, we will have an API token that we must keep in a safe place:

Group alarm

Create a chat group alarm and add the bot in your chat and get chats IDs.

• Add the Telegram bot to the group.
• Get the list of updates for your BOT. Go to the following address in your web browser:

https://api.telegram.org/bot<YourBOTToken>/getUpdates

For example:

https://api.telegram.org/botbot5651130684:AAGqCReqlE97ddjX8Bar4FrYnnS7x/getUpdates

The output of the “chat” object is something like this:

"my_chat_member":{"chat":{"id":-1001423600466,"title":"NODOS CUMULO","type":"supergroup"},"from":{"id":887562799,"is_bot":false,"first_name":"MON \ud83e\uddbe |

You need the chat id.

Now you can test sending a message to the group with a curl:

curl -X POST "https://api.telegram.org/bot5651130684:AAGqCReqlE97ddjX8Bar4FrYnnS7xSJ3x8U/sendMessage" -d "chat_id=-1001423600466&text=Hi how are you bot"

You will see the result in the Telegram chat:

Telegram settings

Open config.yml file and configure Telegram settings:

# Telegram settings
      telegram:
        enabled: yes
        api_key: "5651130684:AAGqCReqlE97ddjX8Bar4FrYnnS7xSJ3x8U" # uses default if blank
        channel: "-1001423600466" # uses default if blank

You must restart tenderduty:

sudo systemctl restart tenderdutyd && sudo journalctl -fu tenderdutyd

Your telegram bot is now ready to send notifications:

Follow OKP4 in Twitter , LinkedIn o Discord