How to install Prometheus and Grafana

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. Grafana is a multi-platform open source analytics and interactive visualization web application. The integration of Prometheus and Grafana offers several significant advantages:

• Robust Monitoring Capabilities
• Support for Diverse Data Sources
• Real-time Monitoring and Alerting
• Ease of Scalability and Customization
• Community Support and Maintenance

Install Prometheus

Prometheus is available as Docker images on Quay.io or Docker Hub. Using container is easy to deploy and manage, especially you have run a lot of services in container on the host. SSH into your node machine and sign in as root.

Prepare configuration and data directory

• Prometheus configuration prometheus.yml is stored in /etc/prometheus directory inside the container.
• Prometheus data is stored in /prometheus directory inside the container.

To run Prometheus container with persistent storage, you can create special directory like:

# mkdir -p /prometheus/{config,data}
# ls -l /prometheus
total 0
drwxr-xr-x 2 root root 6 Mar 26 17:40 config
drwxr-xr-x 2 root root 6 Mar 26 17:40 data

Create your prometheus.yml like:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

And save to /prometheus/config/prometheus.yml

Here we don’t use port 9090, but 8200 as my host has too many containers.

Now you can start container by the following:

docker run -d --name prometheusX --restart unless-stopped -p 8200:9090 \
  -v /prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v /prometheus/data/:/prometheus prom/prometheus

But if you see this error:

ts=2024-03-27T01:49:18.942Z caller=main.go:615 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2024-03-27T01:49:18.942Z caller=main.go:616 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2024-03-27T01:49:18.942Z caller=query_logger.go:93 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"

That means to be failed to access data directory. To fix it, you can do this:

# chmod -R 777 /prometheus/data/
# docker restart prometheus_x

Open the URL http://your-node-ip:8200 on your browser. Your Prometheus container is running well if you can see like:

Install Node Exporter

There is not any exporter configured in Prometheus so no data show there. We can add node_exporter to collect monitor data. Node exporter exposes a wide variety of hardware- and kernel-related metrics.

You can find the latest release of Node exporter from https://github.com/prometheus/node_exporter/releases based on your distribution.

Here we install v1.7.0 on Ubuntu. The Ubuntu is as Node to be monitored by your Prometheus. SSH into your Ubuntu machine and sign in as root. Then you can run the following:

# wget -c https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

# tar xvfz node_exporter-*.*-amd64.tar.gz
node_exporter-1.7.0.linux-amd64/
node_exporter-1.7.0.linux-amd64/LICENSE
node_exporter-1.7.0.linux-amd64/node_exporter
node_exporter-1.7.0.linux-amd64/NOTICE

# cd node_exporter-*.*-amd64
# ./node_exporter
ts=2024-03-27T02:14:17.345Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)"
ts=2024-03-27T02:14:17.345Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)"
ts=2024-03-27T02:14:17.345Z caller=node_exporter.go:195 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2024-03-27T02:14:17.347Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-03-27T02:14:17.347Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-03-27T02:14:17.347Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-03-27T02:14:17.347Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
ts=2024-03-27T02:14:17.347Z caller=node_exporter.go:117 level=info collector=arp
ts=2024-03-27T02:14:17.347Z caller=node_exporter.go:117 level=info collector=bcache

But it is not good to run on your host with root permission. We can customize. The following script can help.

• Create common user/group: node_exporter/node_exporter
• As start service managed by Systemd

#!/bin/bash
# Please make sure node_exporter in /usr/local/bin
# Or anywhere $PATH including

[[ -f /etc/systemd/system/node_exporter.service ]] && {
  echo "Service file found. node_exporter has been deployed."
  exit 1
}

[[ -f /usr/local/bin/node_exporter ]] || {
  echo "node_exporter not found."
  exit 1
}


# create user and group for running node-exporter process
useradd -m node_exporter
groupadd node_exporter
usermod -a -G node_exporter node_exporter
chown node_exporter:node_exporter /usr/local/bin/node_exporter

cat <<EOF > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload || exit 2
systemctl start node_exporter || exit 2

sleep 1 && systemctl status node_exporter

Verify if node exported is running by the following:

# From local
curl http://localhost:9100/metrics

# Or from remote
# Please make sure port 9100 is open
curl http://your-ubuntu-node-ip:9100/metrics

Output like:

 # TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.3807e-05
go_gc_duration_seconds{quantile="0.25"} 2.2453e-05
go_gc_duration_seconds{quantile="0.5"} 3.8102e-05
go_gc_duration_seconds{quantile="0.75"} 4.9015e-05
go_gc_duration_seconds{quantile="1"} 0.000170363
go_gc_duration_seconds_sum 17.558291367
go_gc_duration_seconds_count 423941
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
......

Configure Node Exporter for Prometheus

To configure your node exporter to your Prometheus configuration file, you can append the following to /prometheus/config/prometheus.yml

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "Node monitor"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["your-ubuntu-node-ip:9100"]

In fact, it adds a new job names “Node monitor”. “your-ubuntu-node-ip” is hostname or ip address that is running Node exporter.

Restart your Prometheus container

docker restart prometheus_x

Refresh URL http://your-node-ip:8200 and switch to Status > Targets, you can see data on your Prometheus:

So you have monitored Ubuntu node successfully. Next, we need to view these data gracefully.

Install Grafana

Grafana is available as Docker images on Docker Hub. Here we still deploy Grafana container. Grafana data is stored in /var/lib/grafana directory inside the container. To run Grafana container with persistent storage, you can create special directory like:

mkdir -p /grafana/data

Now you can start container by the following:

docker run -d --name grafanaX --restart unless-stopped -p 3000:3000 \
  -v /grafana/data:/var/lib/grafana \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel, grafana-simple-json-datasource" \
  grafana/grafana-enterprise

“grafana-clock-panel” and “grafana-simple-json-datasource” are plugins of Grafana. Also you can install them later.

Open URL http://your-node-ip:3000 and you can see like this:

Next, you need to add your Prometheus in Grafana.

Configure Prometheus for Grafana

To add the Prometheus data source, go to Home > Connections > Data Sources complete the following steps:

• Under Connections, click Add new connection.
• Enter Prometheus in the search bar.
• Select Prometheus data source.
• Click Create a Prometheus data source in the upper right.

You will be taken to the Settings tab where you will set up your Prometheus configuration.

• Name — The data source name.
• Default — Toggle to select as the default name in dashboard panels.
• URL — The URL of your Prometheus server. Here http://your-node-ip:8200
• Allowed cookies — Specify cookies by name that should be forwarded to the data source.
• Timeout — The HTTP request timeout. This must be in seconds.
• Authentication — No Authentication or your credentials. Here we don’t configure authentication for Prometheus.

Last Save & test.

Thus, we added data source. Next, we will add a dashboard to show the data in time series. Before that, we need to know what is dashboard.

A dashboard is a set of one or more panels organized and arranged into one or more rows. Grafana ships with a variety of panels making it easy to construct the right queries, and customize the visualization so that you can create the perfect dashboard for your need. Each panel can interact with data from any configured Grafana data source.

Here, we use this dashboard:

Node Exporter Full | Grafana Labs

Go to Home > Dashboards, click the button “New” and select “Import”. Copy and paste URL of dashboard. Then click the button “Load”

Give Name, Folder and data source (just added). Then click the button “Import”.

Last, you will see the following cool page. Start to monitor your node!

In summary, the integration of Prometheus and Grafana offers powerful capabilities in monitoring and alerting, assisting users in better managing and maintaining their systems and applications.