A step-by-step guideto setup Prometheus Server for Monitoring

The article contains an instructional Manual for installing the Prometheus server as part of the Observability configuration.

A step-by-step guide to installing the Prometheus server.

Creating Service Users:

sudo useradd --no-create-home --shell /bin/false prometheus

Make the appropriate folders for storing Prometheus files and data before we download the Prometheus binaries. We’ll make a directory in /etc for Prometheus’ configuration files and one in /var/lib for its data in accordance with conventional Linux practices.

 sudo mkdir /etc/prometheus
 sudo mkdir /var/lib/prometheus

Set the Prometheus user as the owner of the user and group accounts for the new directories.

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Downloading Prometheus

Select the latest version of Prometheus https://prometheus.io/download
Make sure you are in the home directory.

cd ~
wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz

Unpack the downloaded archive and Copy the two binaries to the /usr/local/bin directory.

tar xvf prometheus-2.43.0.linux-amd64.tar.gz
sudo cp prometheus-2.43.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.43.0.linux-amd64/promtool /usr/local/bin/

Set the user and group ownership on the binaries to the Prometheus user created.

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

The consoles and console_libraries folders should now be copied to /etc/prometheus.

sudo cp -r prometheus-2.43.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.43.0.linux-amd64/console_libraries /etc/prometheus

The folders should have user and group ownership granted to the Prometheus user.

If the -R option is used, the ownership of the directory’s files will also be set.

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Finally, delete any remaining files from your home directory that are no longer required. (optional)

 rm -rf prometheus-2.43.0.linux-amd64.tar.gz prometheus-2.43.0.linux-amd64      

Setting up Prometheus to monitor resources.

Make a prometheus.yml configuration file in the /etc/prometheus directory using vim or your preferred text editor. For the time being, this file will just include the information necessary to launch Prometheus.

cd /etc/prometheus
sudo vim prometheus.yml

global:
  scrape_interval: 15s
rule_files:
  - alert.rules.yml
​
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:9090']
​
  - job_name: 'node_exporter'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:9100']
​
  - job_name: 'alertmanager'
    static_configs:
      - targets: ['localhost:9093']
​
  - job_name: 'Consul'
    metrics_path: '/v1/agent/metrics'
    static_configs:
      - targets: ['172.31.xx.x:8500']
      ​
  - job_name: 'elasticsearch-cluster'
    static_configs:
      - targets: ['172.31.1.xxx:9114']
​
  - job_name: 'cassandra-cluster'
    static_configs:
      - targets: ['172.31.8.xxx:8080']
​
  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['172.31.4.xxx:15692']
​
  - job_name: 'monitoring-api'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://1zkdg3soq5.execute-api.ap-south-1.amazonaws.com/test
        ​
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Save the file and exit your text editor.

Finally, change the configuration file’s user and group ownership to the Prometheus user.

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Now that the setup is complete, we can launch Prometheus for the first time to test it.

Running Prometheus

Start Prometheus as the Prometheus user and specify the location of the data directory and configuration file.

sudo -u prometheus /usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

Create a service file for Prometheus.

sudo vim /etc/systemd/system/prometheus.service

The configuration file for Prometheus is placed in the /etc/prometheus/prometheus.yml directory, and the service file instructs systemd to execute Prometheus as the Prometheus user and save its data in the /var/lib/prometheus directory.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Finally, save the file and exit the text editor.

To use the newly created service, reload systems.

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus

The command sudo systemctl enable prometheus is used to enable Prometheus as a system service. By executing this command with superuser privileges (sudo), you register Prometheus as a service that will start automatically during system boot.

sudo systemctl enable prometheus

The Prometheus server has been successfully set up and configured, and it is now ready for monitoring and collecting metrics from various targets

Additional information

To start monitoring resources by simply providing tags, you can below configuration:

Add the below syntax in your prometheus.yml file.

- job_name: 'Monitoring-server'
    ec2_sd_configs:
    - region: ap-south-1
      port: 9100
    - region: ap-south-1
      port: 9323
    relabel_configs:
    - source_labels: [__meta_ec2_tag_PrometheusEnabled]
      regex: True
      action: keep
    - source_labels: [__meta_ec2_tag_Name,__meta_ec2_private_ip]
      target_label: instance

Once you modify the prometheus.yml provide access to the Prometheus server to read the tags of the servers.

Create and attach an AWS inline policy to the role which is associated with the Prometheus server.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        }
    ]
}

Provide the following Tag value and key to the resources for monitoring.

PrometheusEnabled = True

Bash Script to install Prometheus server.

#!/bin/bash
# Update the system
#sudo apt-get update
sleep 1
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.42.0/prometheus-2.42.0.linux-amd64.tar.gz
sleep 10
tar xvf prometheus-2.42.0.linux-amd64.tar.gz
sleep 2
sudo cp prometheus-2.42.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.42.0.linux-amd64/promtool /usr/local/bin/
sleep 2
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo cp -r prometheus-2.42.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.42.0.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sleep 5
rm -rf prometheus-2.42.0.linux-amd64.tar.gz prometheus-2.42.0.linux-amd64
sudo touch /etc/prometheus/prometheus.yml
​
echo "global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']" | sudo tee /etc/prometheus/prometheus.yml
cd 
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sleep 2 
sudo touch /etc/systemd/system/prometheus.service
echo "[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
​
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
​
[Install]
WantedBy=multi-user.target" | sudo tee /etc/systemd/system/prometheus.service
sleep 2
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus
exit
echo "prometheus installation Completed!!! "