Custom Monitoring a Fuel node with Grafana and Node Exporter

Table of contents

· Introduction
· Available Resources
· Metrics environment setup
∘ Step 1: Prepare the environment
∘ Step 2: Setting up the Node Exporter service
∘ Step 3: Service and Timer Configuration
∘ Step 4: Verification

Introduction

This document provides a comprehensive guide on implementing a custom metrics monitoring system for a Fuel node using Prometheus and Node Exporter. Node Exporter is a vital tool that allows for the collection of operating system and server hardware metrics and can be configured to expose metrics specific to the Fuel node.

Throughout this guide, you will learn how to set up Node Exporter to capture and expose these metrics and automate the process using scripts and systemd services. This system will give you detailed insights into the performance and status of the Fuel node, facilitating more efficient monitoring and maintenance of the system.

Available Resources

https://github.com/Cumulo-pro/FuelTools/blob/main/fuel_metrics/install_fuel_metrics.md

Implementation and Configuration of Custom Metrics Monitoring

https://github.com/Cumulo-pro/FuelTools/blob/main/fuel_metrics/fuelmetrics.sh

Bash script designed to collect various metrics related to the fuel service and save this data in a metrics file in Prometheus format.

https://github.com/Cumulo-pro/FuelTools/blob/main/fuel_metrics/Fuel%20metrics-1718733581399.json

This resource provides a JSON configuration file for a Grafana dashboard. The file includes all the necessary definitions and settings to visualize Fuel node metrics in Grafana, making it easy to create informative panels and interactive graphs for real-time monitoring.

https://grafana.com/grafana/dashboards/21370-fuel-metrics/

Official Grafana dashboard for Fuel metrics, which can be imported directly into Grafana for monitoring fuel service metrics.

This resource provides a JSON configuration file for a Grafana dashboard. The file includes all the necessary definitions and settings to visualize Fuel node metrics in Grafana, making it easy to create informative panels and interactive graphs for real-time monitoring.

Metrics environment setup

Node Exporter is a Prometheus tool that collects operating system and server hardware metrics. However, it can also be used to expose custom metrics.

Prerequisites:

• Install node_exporter on your server

GitHub - prometheus/node_exporter: Exporter for machine metrics

• Install Grafana Dashboard

GitHub - prometheus/node_exporter: Exporter for machine metrics

Step 1: Prepare the environment

Create a metrics directory to store metrics scripts.

sudo mkdir -p /usr/local/metrics

Download the metrics script fuelmetrics.sh

This is a bash script that extracts the desired metrics from the bridge node and sends them to Prometheus. The script is not intrusive it just executes a series of CLI commands and sends the metrics to node_exporter_metrics.prom file, these metrics will be picked up by node_exporter to expose them in Prometheus.

sudo wget https://raw.githubusercontent.com/Cumulo-pro/FuelTools/main/fuel_metrics/fuelmetrics.sh -O /usr/local/metrics/fuelmetrics.sh

Provides the necessary permissions to the script

sudo chmod +x /usr/local/metrics/fuelmetrics.sh

Step 2: Setting up the Node Exporter service

Edit the Node Exporter service file (already installed and running on your server):

/etc/systemd/system/node_exporter.service

Add the option — collector.textfile.directory to specify the directory where the metrics generated by the script will be stored:

sudo sed -i 's|ExecStart=/usr/local/bin/node_exporter.*|ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/usr/local/metrics|' /etc/systemd/system/node_exporter.service

Reload systemd: Once the file has been saved, reload the systemd daemon so that it recognises the changes:

sudo systemctl daemon-reload

Restart the service:

sudo systemctl restart node_exporter

Provides the necessary permissions for node_exporter:

sudo chown -R node_exporter:node_exporter /usr/local/metrics
sudo chmod -R 755 /usr/local/metrics

Step 3: Service and Timer Configuration

Create a service file and timer to run the script periodically /etc/systemd/system/ful_metrics.service:

sudo tee /etc/systemd/system/fuel_metrics.service > /dev/null << EOF
[Unit]
Description=Fuel metrics update
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/metrics/fuelmetrics.sh

[Install]
WantedBy=multi-user.target
EOF

Enable and start services:

sudo systemctl daemon-reload
sudo systemctl enable fuel_metrics.service
sudo systemctl start fuel_metrics.service

Create the timer /etc/systemd/system/fuel_metrics.timer:

sudo tee /etc/systemd/system/fuel_metrics.timer > /dev/null << EOF
[Unit]
Description=Timer for bridge metrics update

[Timer]
OnUnitActiveSec=6s
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable and start services:

sudo systemctl daemon-reload
sudo systemctl enable --now fuel_metrics.timer

Step 4: Verification

Check that the Node Exporter, fuel_metrics and the timer are working correctly:

sudo systemctl status node_exporter
sudo systemctl status fuel_metrics.service
sudo systemctl status fuel_metrics.timer

Check metrics:

curl http://localhost:9100/metrics | grep fuel_height

Once we have completed these steps, we can access Grafana and perform the necessary steps to visualise our customised Fuel node metrics.

Twitter | Medium | LinkedIn | Discord | Telegram | cumulo.pro