Keeping up with your Oasis node — Monitoring with PANIC Oasis

PANIC for Oasis is a lightweight yet powerful open source monitoring and alerting solution for Oasis nodes by Simply VC. The tool was built with user friendliness in mind, without excluding cool and useful features like phone calls for critical alerts and Telegram commands for increased control over your alerter.

What we’ll need:

1. Node exporter on each node to export system statistics.
2. An Oasis API server agent running beside each node.
3. PANIC for Oasis running on a separate VM.

Demo of setting up PANIC for Oasis

Set up dependencies on both Oasis and PANIC hosts

• Python3 and Python-pip (required on both Oasis node and PANIC hosts)

$ sudo apt-get update
$ sudo apt-get install python3.6 -y
$ sudo apt-get install python3-pip -y
$ sudo pip3 install pipenv

• Docker (required on both Oasis node and PANIC hosts)

$ curl -sSL https://get.docker.com/ | sh

• Docker Compose (required on PANIC host)

$ sudo apt install docker-compose -y

• Installing Node Exporter (required on Oasis node hosts)

Node exporter is used to export metrics related to each system’s hardware metrics, for alerting and monitoring of them through PANIC.

Create a Node Exporter user for running the exporter:

sudo useradd --no-create-home --shell /bin/false node_exporter

Download and extract the latest version of Node Exporter:

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar -xzvf node_exporter-1.0.1.darwin-amd64.tar.gz
sudo cp node_exporter-1.0.1.darwin-amd64/node_exporter /usr/local/bin/

Give the Node Exporter user ownership of the executable:

sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Perform some cleanup and create a Node Exporter service with the below contents:

sudo rm node_exporter-1.0.1.darwin-amd64 -rf
sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
 
[Install]
WantedBy=multi-user.target

Reload systemctl services list, start the service and enable it to have it start on system restart:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

CURL your node exporter port to confirm that it is not erroring:

curl http://127.0.0.1:9100/metrics

Install and Run the Oasis API Server

In order for the API to be able to run correctly, Prometheus metrics should be enabled in the Oasis node’s configuration file config.yml in the /serverdir/etc/ directory which was set during the installation of the Oasis node. To enable the metrics append this to the end of the file and restart your node:

metrics:
    mode: pull
    address: 0.0.0.0:9090

Change :9090 to the port you want prometheus to be exposed at.

Configuring the API

Configuring the API on the host of your Oasis VM involves setting up three config files.

Start off by cloning this repository and navigating into it:

git clone https://github.com/SimplyVC/oasis_api_server
cd oasis_api_server

You can run the Python script run_setup.py which will guide you through the configuration process, and ask you to enter the values one by one. This can be done by running the following, starting from the root project directory, oasis_api_server . Please note that sentry data does not need to be set up.

pipenv sync
pipenv run python run_setup.py

Installing the API

Run the following commands to build the Docker image:

cd oasis_api_server
docker build -t simplyvc/oasis_api_server:latest .

Running the Docker Image

Now that the Docker image is on your machine, and you have written configurations for it, you can run it as follows, where:

CONFIG_DIR for eg. /oasis_api_server/config

INTERNAL_SOCK_DIR for eg. /serverdir/node/internal.sock

PATH_IN_NODE_CONFIG for eg. /serverdir/node/internal.sock

Now that the Docker image is on your machine and you have written configuration for it, you can run it as follows:

docker run --network="host" -p 127.0.0.1:8080:8080 --mount type=bind,source=<CONFIG_DIR>,target="/app/config/" --mount type=bind,source=<INTERNAL_SOCK_DIR>,target=<PATH_IN_NODE_CONFIG> -d simplyvc/oasis_api_server:latest

Confirming the API Works

If you wish to make sure that the API is running, the following should return {"result":"pong"}:

curl -X GET http://localhost:8080/api/ping

If you wish to check the API’s connection to a node, you can run the following for some node <NODE>, being the identifier of the node to be tested.

curl -X GET http://localhost:8080/api/pingnode?name=<NODE>

Running PANIC

Start by cloning the panic_oasis repository and navigating into it:

git clone https://github.com/SimplyVC/panic_oasis
cd panic_oasis

At any stage, you can use the command docker ps to confirm that the installed component (Redis, MongoDB, the UI and alerter) is running

Now that you have it, set up your configuration files for PANIC’s components, including Redis, Mongo and UI authentication, togeter with the nodes to be monitored:

pipenv sync
pipenv run python run_alerter_setup.py        
pipenv run python run_ui_setup.py

Turning on the alerter

To deploy Mongo, Redis, PANIC UI and the PANIC alerter, use the following command inside the panic_oasis directory:

docker-compose up -d --build

You should now be able to access the UI on a browser with your authentication details.

Stay tuned to Simply VC for more Oasis related guides and documentation:

• Website: https://simply-vc.com.mt
• Twitter: https://twitter.com/simply_vc
• Telegram: https://t.me/SimplyVC
• PANIC for Polkadot: https://github.com/simplyVC/panic_polkadot
• PANIC for Cosmos: https://github.com/simplyVC/panic_cosmos