Grafana-Stack: Building a Comprehensive Monitoring Solution for Docker

4 min readOct 7, 2023

Introduction

In the fast-paced world of containerization, monitoring your Docker environment is essential for maintaining performance, diagnosing issues, and ensuring smooth operations. Fortunately, there’s a powerful open-source monitoring stack available that combines Prometheus, Grafana, NodeExporter, AlertManager, and cAdvisor to provide comprehensive insights into your Docker hosts, containers, and services.

In this step-by-step guide, we’ll walk you through the installation and setup of the Grafana-Stack monitoring solution. By the end of this tutorial, you’ll have a fully functional monitoring system in place, allowing you to keep a close eye on the health and performance of your Docker infrastructure.

Let’s get started!

Installation

To begin, let’s install the Grafana-Stack on your Docker host. Follow these simple steps:

Step 1: Clone the Repository

First, clone the Grafana-Stack repository to your Docker host using the following command:

git clone https://github.com/t0mer/grafana-stack
cd grafana-stack

Step 2: Start the Containers

Next, start the Grafana-Stack containers by running the following command:

ADMIN_USER='admin' ADMIN_PASSWORD='admin' docker-compose up -d

This command will initiate various containers, including Prometheus, AlertManager, Grafana, and more, to form your monitoring stack.

Setup Grafana

Now that the Grafana-Stack containers are up and running, let’s configure Grafana to visualize metrics and manage alerts.

Access Grafana Web Interface

Open your web browser and navigate to http://<host-ip>:3000. You can log in with the default credentials:

Username: admin
Password: admin

You can change these credentials by modifying the ADMIN_USER and ADMIN_PASSWORD environment variables in the docker-compose up command or directly in the compose file.

Configure Grafana

Grafana comes preconfigured with dashboards and Prometheus as the default data source. The configuration can be adjusted by editing the compose file and providing a config file, as shown below:

grafana:
  image: grafana/grafana:7.2.0
  env_file:
    - env

The config file format should resemble the following:

GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=changeme
GF_USERS_ALLOW_SIGN_UP=false

Remember, if you change the password, be sure to remove the old password entry to apply the change.

Explore Dashboards

Grafana comes with preconfigured dashboards. Two key dashboards are included:

Docker Host Dashboard

This dashboard provides insights into your server’s resource usage, including uptime, CPU usage, memory, storage, and more.

Docker Containers Dashboard

The Containers Dashboard allows you to monitor running containers, including CPU load, memory usage, IO, and network usage.

Customize Storage Metrics

For specific storage metrics, such as Free Storage, you may need to specify the fstype in Grafana graph requests. Locate the relevant configuration file and update it according to your filesystem type.

Define Alerts

To effectively monitor your Docker environment, you can define alerts tailored to your needs.

Set Up Alerts

Alerts can be configured within the alert.rules file in the Grafana-Stack repository. There are three alert groups:

Monitoring services alerts
Docker Host alerts
Docker Containers alerts

You can modify these rules to trigger alerts based on specific conditions, such as service downtime, high CPU load, or low storage space.

Reload Alert Rules

After modifying the alert rules, you can reload them by making an HTTP POST call to Prometheus.

curl -X POST http://admin:admin@<host-ip>:9090/-/reload

This ensures that your new alert rules take effect.

Setup Alerting

The AlertManager service handles alerts sent by Prometheus and can notify you via various channels, including email, Slack, or custom webhooks.

Configure AlertManager

AlertManager’s notification receivers can be configured in the alertmanager/config.yml file in the Grafana-Stack repository. To receive alerts via Slack, create a custom integration and provide the required information in the config file.

Sending Metrics to the Pushgateway

The Pushgateway is used to collect data from batch jobs or services. To push data, follow these steps.

Push Metrics to PushgatewayExecute the following command to push metrics to the Pushgateway

echo "some_metric 3.14" | curl --data-binary @- http://user:password@localhost:9091/metrics/job/some_job

Replace user:password with your specified user and password (default: admin:admin).

Summary

Congratulations! You’ve successfully set up the Grafana-Stack monitoring solution for your Docker environment. With this comprehensive monitoring stack in place, you can proactively manage your Docker infrastructure, detect issues, and ensure smooth operations.

This guide is based on the fantastic work of stefanprodan and the dockprom project. Special thanks to them for their contributions to the Docker monitoring community. Enjoy monitoring your Docker environment with Grafana-Stack!