Monitoring Servers and Docker Containers using Prometheus with Grafana

Docker Containers using Prometheus with Grafana

Cypik.com

Introduction

Infrastructure monitoring is the basis for application performance management. The underlying system’s availability and health must be maximized continually. To achieve this, one has to monitor system metrics like CPU, memory, network, and disk. Response time lag, if any, must be addressed swiftly. Here we’ll take a look at how to monitor servers (and even Docker containers running inside the server) using Grafana, Prometheus, Node Exporter, and CAdvisor.

Core Components

Grafana: Database for Analytics & monitoring solution

Prometheus: Event monitoring and alerting

Node-Exporter: Monitoring Linux host metrics

CAdvisor: Monitoring metrics for the running Containers.

Grafana — Analytics & monitoring solution for database

Grafana equips users to query, visualize, and monitor metrics, no matter where the underlying data is stored. With Grafana, one can also set alerts for metrics that require attention, apart from creating, exploring, and sharing dashboards with their team and fostering a data-driven culture.

Prometheus: event monitoring and alerting

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open-source project that is maintained independently of any company.

Node Exporter — Monitoring Linux host metrics

Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable metric collectors. It allows measuring various machine resources, such as memory, disk, and CPU utilization

CAdvisor — Monitoring metrics for the running Containers.

It Stands for Container Advisor and is used to aggregate and process all the metrics for the running Containers

Prerequisite,

1. A Linux machine
2. Docker Installed
3. Docker Compose is installed.

sudo apt install docker.io -y
sudo usermod -aG docker $USER && newgrp docker
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
sudo docker network create monitor-net

So Let’s get started.

Login to your Linux machine, update the repository, and install Docker and Docker Compose. To Update the Repository,

Create a Directory, say monitoring

suresh@suresh:~ mkdir monitoring 
suresh@suresh:~ cd monitoring/
suresh@suresh:~ vim docker-compose.yml

Now, create a Docker Compose file for Prometheus. You also need to create a Prometheus configuration file, prometheus.yml, in the Docker Compose file for Prometheus as below.

Note: We will keep on extending the same docker file as we move forward to install other components.

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_db:/var/lib/prometheus
      - ./alert.rules:/etc/prometheus/alert.rules
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.route-prefix=/'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    ports:
      - '9090:9090'
    networks:
      - monitor-net

Create a Prometheus configuration file and paste the configuration as below.

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'suresh-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['server_ip:9090'] ## IP Address of the local host

The compose file consists of two-volume mappings to the container. One is the Prometheus configuration and the other (prometheus_db) is to store the Prometheus database locally. Now run Docker Compose.

suresh@suresh:~/monitoring$ mkdir prometheus_db
suresh@suresh:~/monitoring$ sudo docker-compose up -d

Access Prometheus using the IP and port, and you will see the Prometheus UI, http://ip_address:9090

Now we will setup the Node Exporter. It is one of the best components used along with Prometheus to capture metrics from the server where Prometheus is running. It Captures all hardware and kernel-related metrics like CPU, Memory, Disk, Disk Read/Write, etc.

To Install the Node exporter, simply append the docker-compose.yml file and prometheous.yml file as below.

node-exporter:
  image: prom/node-exporter
  ports:
    - '9100:9100'

Append the prometheus.yml as below.

- job_name: 'node-exporter' 
  static_configs: 
    - targets: ['server_ip:9100']

So now the composite docker-compose file will look like below:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_db:/var/lib/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - '9090:9090'
  node-exporter:
  image: prom/node-exporter
  ports:
    - '9100:9100'

And prometheus.YML will look like below:

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'suresh-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['server_ip:9090'] ## IP Address of the local host

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['server_ip:9100']

Now Create the Node Exporter Container and Restart the Prometheus container using the below commands

suresh@suresh:~/monitoring$ sudo docker-compose start node-exporter
suresh@suresh:~/monitoring$ sudo docker-compose restart prometheus

OR, You can simply compose up and down.

suresh@suresh:~/monitoring$ sudo docker-compose down
suresh@suresh:~/monitoring$ sudo docker-compose up -d

Now that you see the Targets in Prometheus, you will see node exporter as well as a target.

Now, set up CAdvisor; for this, append the docker Composer with below code.

cadvisor:
    image: google/cadvisor:latest
    ports:
      - '8080:8080'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

Also, Append the prometheus.yml with a bit-coded yml code. We are actually adding the CAdvisor service to Prometheus configuration.

  - job_name: 'cAdvisor'
    static_configs:
      - targets: ['server_ip:8080']

Access CAdvisor from the URL, http://IP_Address:8080/docker/

Eventually, we will set up the grafana, where we will be using Prometheus as a data source. We can have a better Dashboard in grafana for the metrics visualization.

Append the code in the above docker compose and restart.

  grafana:
    image: grafana/grafana
    user: "1000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secure_pass
    volumes:
      - ./grafana_db:/var/lib/grafana
    depends_on:
      - prometheus
    ports:
      - '3000:3000'

Access Grafana Url from the 3000 port; the default user will be admin, and the password you set in the compose file.

Example: docker-compose.yml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_db:/var/lib/prometheus
      - ./alert.rules:/etc/prometheus/alert.rules
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.route-prefix=/'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    ports:
      - '9090:9090'
    networks:
      - monitor-net

  cadvisor:
    image: google/cadvisor:latest
    ports:
      - '8080:8080'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  node-exporter:
    image: prom/node-exporter
    ports:
      - '9100:9100'

  grafana:
    image: grafana/grafana
    user: "1000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secure_pass
    volumes:
      - ./grafana_db:/var/lib/grafana
    depends_on:
      - prometheus
    ports:
      - '3000:3000'
    networks:
      - monitor-net

networks:
  monitor-net:
    driver: bridge

volumes:
  prometheus_db:
  grafana_db:

Example: prometheus.yml

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'suresh-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['server_ip:9090'] ## IP Address of the local host

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['server_ip:9100']

  - job_name: 'cAdvisor'
    static_configs:
      - targets: ['server_ip:8080']

Grafana container not running; use commands

sudo rm -rf ./grafana_db
sudo mkdir ./grafana_db
sudo chmod -R 777 ./grafana_db

sudo docker-compose up -d

Enjoy it! 🍻 That’s It; we are done...

For seamless Cloud Management incorporating DevOps as the core of the methodology, reach out to us at info@cypik.com

Cypik

About the author:

My name is Suresh Yadav, and I am an experienced Linux enthusiast and DevOps engineer. I’m passionate about automating and streamlining development processes, and currently, I work as a DevOps Engineer at Cypik. I specialize in cloud technologies, with a focus on the Google Cloud Platform (GCP).