Open in app

Sign in

Write

Sign in

Monitoring Servers and Docker Containers using Elasticsearch with Grafana

Prakash Iyyanarappan
Skedler
Published in
5 min readFeb 1, 2020

Introduction

Infrastructure monitoring is the basis for application performance management. The underlying system’s availability and health must be maximized continually. To achieve this, one has to monitor the system metrics like CPU, memory, network, and disk. Response time lag, if any must be addressed swiftly. Here we’ll take a look at how to Monitor servers (and even Docker Containers running inside the Server) using Grafana, Elasticsearch, Metricbeat, and Skedler Reports.

Core Components

Grafana-Analytics & monitoring solution for database

Elasticsearch-Ingest and index logs

Metricbeat-Lightweight shipper for metrics

Skedler Reports -Automate actionable reports

Grafana — Analytics & monitoring solution for database

Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data-driven culture.

Elasticsearch-Ingest and index logs

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

Metricbeat — Lightweight shipper for metrics

Collect metrics from your systems and services. From CPU to memory, Redis to NGINX, and much more, Metricbeat is a lightweight way to send system and service statistics.

Skedler Reports — Automate actionable reports

Skedler offers the most powerful, flexible and easy-to-use data monitoring solution that companies use to exceed customer SLAs, achieve compliance, and empower internal IT and business leaders.

Prerequisites

  1. A Linux machine
  2. Docker Installed
  3. Docker Compose Installed
ubuntu@guidanz:~$ mkdir monitoring
ubuntu@guidanz:~$ cd monitoring/
ubuntu@guidanz:~$ vim docker-compose.yml

Now, create a Docker Compose file for Elasticsearch, and an Elasticsearch configuration file, elasticsearch.yml. Docker Compose file for Elasticsearch is below.

version: "2.1"
services:#Elasticsearch container
  elasticsearch:
    container_name: elasticsearch
    hostname: elasticsearch
    image: "docker.elastic.co/elasticsearch/elasticsearch:latest"
    logging:
      options:
        max-file: "3"
        max-size: "50m"
    environment:
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms${ES_JVM_HEAP} -Xmx${ES_JVM_HEAP}"
    mem_limit: 1g
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - esdata:/usr/share/elasticsearch/data
    ports: ['9200:9200']
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    networks: ['stack']volumes:
  esdata:
    driver: local
networks: guidanz

Note: We will keep on extending the same docker file as we will move ahead to install other components.

Create an Elasticsearch configuration file elasticsearch.yml and paste the config as below.

cluster.name: guidanz-stack-cluster
node.name: node-1
network.host: 0.0.0.0
path.data: /usr/share/elasticsearch/data
http.port: 9200
xpack.monitoring.enabled: true
http.cors.enabled: true
http.cors.allow-origin: "*"
http.max_header_size: 16kb

Now run the docker-compose.

ubuntu@guidanz:~/monitoring$ docker-compose up -d

Access Elasticsearch using the IP and Port and you will see the Elasticsearch UI.

http://ip_address:9200

Now we will setup the Metricbeat. It is one of the best components used along with the Elasticsearch to capture metrics from the server where the Elasticsearch is running. It captures all hardware and kernel-related metrics like system-level CPU usage, memory, file system, disk IO, and network IO statistics, as well as top-like statistics for each of the processes running on your systems.

To install the Metricbeat, simply append the docker-compose.yml file, metricbeat.yml, and modules.d file as below.

metricbeat:
    container_name: metricbeat
    hostname: metricbeat
    user: root #To read the docker socket
    image: docker.elastic.co/beats/metricbeat:latest
    logging:
      options:
        max-file: "3"
        max-size: "50m"
    volumes:
      #Mount the metricbeat configuration so users can make edits.
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
      #Mount the modules.d directory into the container. This allows user to potentially make changes to the modules and they will be dynamically loaded.
      - ./modules.d/:/usr/share/metricbeat/modules.d/
      #The commented sections below enable Metricbeat to monitor the Docker host rather than the Metricbeat container. These are used by the system module.
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      #Allows us to report on docker from the hosts information.
      - /var/run/docker.sock:/var/run/docker.sock
      #We mount the host filesystem so we can report on disk usage with the system module.
      - /:/hostfs:ro
    command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
    networks: ['stack']
    restart: on-failure
#    environment:
#      - "MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}"
    depends_on:
      elasticsearch:  { condition: service_healthy }

Append the metricbeat.yml as below,

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 5s
  reload.enabled: trueprocessors:
- add_docker_metadata: ~monitoring.enabled: true
setup.ilm.enabled: falseoutput.elasticsearch:
  hosts: ["elasticsearch:9200"]
  
logging.to_files: falsesetup:
  kibana.host: "kibana:5601"
  dashboards.enabled: true

The compose file consists of the volume mappings to the container. One is the metricbeat configuration and the second one (modules.d) is to mount the modules.d directory into the container. This allows users to potentially make changes to the modules and they will be dynamically loaded. Now run the docker-compose.

ubuntu@guidanz:~/monitoring$ mkdir modules.d

Append the system.yml as below inside the module.d folder,

- module: system
  metricsets:
    - core
    - cpu
    - load
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
    - socket
  enabled: true
  period: 5s
  processes: ['.*']
  cpu_ticks: true
  process.cgroups.enabled: true
  process.include_top_n:
    enabled: true
    by_cpu: 20
    by_memory: 20

So now the composite docker-compose file will look like below,

version: "2.1"
services:#Elasticsearch container
  elasticsearch:
    container_name: elasticsearch
    hostname: elasticsearch
    image: "docker.elastic.co/elasticsearch/elasticsearch:latest"
    logging:
      options:
        max-file: "3"
        max-size: "50m"
    environment:
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms${ES_JVM_HEAP} -Xmx${ES_JVM_HEAP}"
    mem_limit: 1g
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - esdata:/usr/share/elasticsearch/data
    ports: ['9200:9200']
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    networks: ['stack']metricbeat:
    container_name: metricbeat
    hostname: metricbeat
    user: root #To read the docker socket
    image: docker.elastic.co/beats/metricbeat:latest
    logging:
      options:
        max-file: "3"
        max-size: "50m"
    volumes:
      #Mount the metricbeat configuration so users can make edits.
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
      #Mount the modules.d directory into the container. This allows user to potentially make changes to the modules and they will be dynamically loaded.
      - ./modules.d/:/usr/share/metricbeat/modules.d/
      #The commented sections below enable Metricbeat to monitor the Docker host rather than the Metricbeat container. These are used by the system module.
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      #Allows us to report on docker from the hosts information.
      - /var/run/docker.sock:/var/run/docker.sock
      #We mount the host filesystem so we can report on disk usage with the system module.
      - /:/hostfs:ro
    command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
    networks: ['stack']
    restart: on-failure
#    environment:
#      - "MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}"
    depends_on:
      elasticsearch:  { condition: service_healthy }volumes:
  esdata:
    driver: local
networks: guidanz

You can simply do compose up and down.

ubuntu@guidanz:~/monitoring$ docker-compose down 
ubuntu@guidanz:~/monitoring$ docker-compose up -d

Now take a look at the Targets in Elasticsearch. You will notice Metricbeat as well as a target.

Now let’s set up Grafana, where we will be using Elasticsearch as a data source. We can have a better Dashboard in grafana for the metrics visualization.

Append the code in the above docker compose and restart.

grafana:
 image: grafana/grafana
 user: "1000"
 environment:
   - GF_SECURITY_ADMIN_PASSWORD=secure_pass
 volumes:
   - ./grafana_db:/var/lib/grafana
 depends_on:
   - Elasticsearch
 ports:
   - '3000:3000'

Access Grafana UI from 3000 port, default user will be admin and the password you set in the compose file.

Now eventually we will set up the Skedler Reports, where we will be using Grafana as a data source. Skedler offers a simple and easy to add reporting and alerting solution for Elastic Stack and Grafana. Please review the documentation to install Skedler.

Now, Setup Skedler Reports, for this append the docker compose with the below code.

reports:
   image: skedler/reports:latest
   container_name: reports
   privileged: true
   cap_add:
     - SYS_ADMIN
   volumes:
     - /sys/fs/cgroup:/sys/fs/cgroup:ro
     - reportdata:/var/lib/skedler
     - ./reporting.yml:/opt/skedler/config/reporting.yml
   ports:
     - 3001:3001

Access Skedler Reports from the URL, http://IP_Address:3001

Grafana
Elasticsearch
Skedler Reports
Infrastructure
Docker Compose

--

--

Prakash Iyyanarappan
Skedler

Written by Prakash Iyyanarappan

66 Followers
Writer for

Senior Technical Lead at EY

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams