Monitoring Servers and Docker Containers using Prometheus with Grafana

Introduction

Infrastructure monitoring is the basis for application performance management. The underlying system’s availability and health must be maximized continually. To achieve this, one has to monitor the system metrics like CPU, memory, network, and disk. Response time lag, if any must be addressed swiftly. Here we'll take a look at how to Monitor servers (and even Docker Containers running inside the Server) using Grafana, Prometheus, Node Exporter, CAdvisor and Skedler Reports.

Core Components

Grafana- Database for Analytics & monitoring solution

Prometheus- Event monitoring and alerting

Node-Exporter- Monitoring Linux host metrics

Wmi-Exporter- Monitoring Windows host metrics

CAdvisor- Monitoring metrics for the running Containers.

Skedler Reports -Automating actionable reports

Grafana — Analytics & monitoring solution for database

Grafana equips users to query, visualize, and monitor metrics, no matter where the underlying data is stored. With Grafana, one can also set alerts for metrics that require attention, apart from creating, exploring, and sharing dashboards with their team and fostering a data-driven culture.

Prometheus — Event monitoring and alerting

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.

Node Exporter — Monitoring Linux host metrics

Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable metric collectors. It allows measuring various machine resources such as memory, disk, and CPU utilization

WMI Exporter — Monitoring Windows host metrics

Prometheus exporter for Windows machines, using the WMI (Windows Management Instrumentation).

CAdvisor — Monitoring metrics for the running Containers.

It Stands for Container Advisor and used to aggregate and process all the metrics for the running Containers

Skedler Reports — Automate actionable reports

Skedler offers the most powerful, flexible and easy-to-use data monitoring solution that companies use to exceed customer SLAs, achieve compliance, and empower internal IT and business leaders.

Prerequisite,

1. A Linux machine
2. Docker Installed
3. Docker Compose Installed

So Let’s get started.

Login to your Linux machine, Update the repository and Install Docker and Docker Compose. To Update the Repository,

Create a Directory, say monitoring

ubuntu@guidanz:~$ mkdir monitoring 
ubuntu@guidanz:~$ cd monitoring/
ubuntu@guidanz:~$ vim docker-compose.yml

Now, create a Docker Compose file for Prometheus, You also need to create a Prometheus configuration file, prometheus.yml Docker Compose file for Prometheus as below,

Note: We will keep on extending the same docker file as we move forward to install other components.

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_db:/var/lib/prometheus
      - ./prometheus_db:/prometheus
      - ./prometheus_db:/etc/prometheus
      - ./alert.rules:/etc/prometheus/alert.rules
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.route-prefix=/'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    ports:
      - '9090:9090'
    networks:
      - monitor-net

Create a Prometheus configuration file and paste the config as below.

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'prakash-monitor'
scrape_configs:
  - job_name: 'prometheus' 
    static_configs: 
      - targets: ['monitoring.guidanz.com:9090'] ## IP Address of the localhost

The compose file consists of two volume mappings to the container. One is the Prometheus configuration and the other one (prometheus_db) is to store the Prometheus database locally. Now run the docker-compose.

ubuntu@guidanz:~/monitoring$ mkdir prometheus_db
ubuntu@guidanz:~/monitoring$ docker-compose up -d

Access Prometheus using the IP and Port and you will see the Prometheus UI, http://ip_address:9090

Now we will setup the Node Exporter. It is one of the best components used along with the Prometheus to capture metrics from the server where the Prometheus is running. It Captures all hardware and kernel-related metrics like CPU, Memory, Disk, Disk Read/Write, etc.

To Install the Node exporter, simply append the docker-compose.ymlfile and prometheous.yml file as below.

node-exporter:
  image: prom/node-exporter
  ports:
    - '9100:9100'

Append the prometheus.yml as below,

- job_name: 'node-exporter' 
  static_configs: 
    - targets: ['monitoring.guidanz.com:9100']

So now the Composite docker-compose file will looks like below,

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_db:/var/lib/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - '9090:9090'
  node-exporter:
  image: prom/node-exporter
  ports:
    - '9100:9100'

And prometheus.yml will look like below,

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'devopsage-monitor'
scrape_configs:
  - job_name: 'prometheus' 
    static_configs: 
      - targets: ['monitoring.guidanz.com:9090']
  - job_name: 'node-exporter' 
    static_configs: 
      - targets: ['monitoring.guidanz.com:9100']

Now Create the Node Exporter Container and Restart the Prometheus container using the below commands

ubuntu@guidanz:~/monitoring$ docker-compose start node-exporter
ubuntu@guidanz:~/monitoring$ docker-compose restart prometheus

OR, You can Simply do compose up and down.

ubuntu@guidanz:~/monitoring$ docker-compose down
ubuntu@guidanz:~/monitoring$ docker-compose up -d

Now See the Targets in Prometheus, you will see node exporter as well as a target.

Now, Setup CAdvisor, for this append the docker compose with below code.

cadvisor:
  image: google/cadvisor:latest
  ports:
    - '8080:8080'
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

Also, Append the prometheus.yml with a bit code yml code. We are actually adding the CAdvisor service in Prometheus configuration.

- job_name: 'cAdvisor' 
  static_configs:
    - targets: ['monitoring.guidanz.com:8080']

Access CAdvisor from the URL, http://IP_Address:8080/docker/

Now eventually we will set up the grafana, where we will be using Prometheus as a data source. We can have a better Dashboard in grafana for the metrics visualization.

Append the code in the above docker compose and restart.

grafana:
  image: grafana/grafana
  user: "1000"
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=secure_pass
  volumes:
    - ./grafana_db:/var/lib/grafana
  depends_on:
    - prometheus
  ports:
    - '3000:3000'

Access grafana UI from 3000 port, default user will be admin and the password you set in the compose file.

Now eventually we will set up the Skedler Reports, where we will be using Grafana as a data source. Skedler offers a simple and easy to add reporting and alerting solution for Elastic Stack and Grafana. Please review the documentation to install Skedler Reports.

Now, Setup Skedler Reports, for this append the docker compose with the below code.

  reports:
    image: skedler/reports:latest
    container_name: reports
    privileged: true
    cap_add:
      - SYS_ADMIN
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
      - reportdata:/var/lib/skedler
      - ./reporting.yml:/opt/skedler/config/reporting.yml
    ports:
      - 3001:3001
    

Access Skedler Reports from the URL, http://IP_Address:3001