Configuring Grafana and Prometheus to Monitor my Docker Development Environment

I’ve recently built a development environment for building container-based Python applications and deployed Jenkins to it for CI/CD pipeline configuration, and would like to add a monitoring system to get deeper insights into host/container resource usage.

Previously I’ve deployed a bare-metal Grafana/InfluxDB solution, but while completing A Cloud Guru’s Learn Docker By Doing labs, one lab used Prometheus for metric gathering, rather than Influx. I liked that Prometheus uses pull-based collection, rather than the agent-based push style that Influx uses, and have decided to use Prometheus for this build.

The steps below are how I have configured my pre-existing Docker host for monitoring (should a reader not have a Docker host to deploy to, follow this articleto the Docker Compose section)

Steps Taken

Prepare Docker Host for Metric Exporting

• sudo vim /etc/docker/daemon.json add configuration for the Docker daemon by adding the following text to the file:

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

• sudo systemctl restart docker reload the Docker service to pass through daemon config changes
• firewall-cmd --zone=public --add-port=9323/tcp add the relevant firewall port for Prometheus

Configure Prometheus

I’ve added a section to my python playground repository, which has configuration files for the infrastructure that I am currently practicing deploying Python applications to.

• cd infrastructure/docker && touch monitoring && cd monitoring to create the files for this project and change working directory
• touch prometheus.yml to create the YAML configuration file for Prometheus, and enter the following configuration:

scrape_configs:
  - job_name: prometheus
    scrape_interval: 5s
    static_configs:
    - targets:
      - prometheus:9090
      - node-exporter:9100
      - pushgateway:9091
      - cadvisor:8080

  - job_name: docker
    scrape_interval: 5s
    static_configs:
    - targets:
      - <DOCKER_HOST_IP_ADDRESS>:9323

Create Docker Compose Template

• touch docker-compose.yml to create a Docker Compose file and enter the following:

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
	restart: unless-stopped
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
	restart: unless-stopped
    ports:
      - 8181:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  pushgateway:
    image: prom/pushgateway
    container_name: pushgateway
	restart: unless-stopped
    ports:
      - 9091:9091
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    expose:
      - 9100
  grafana:
    image: grafana/grafana
    container_name: grafana
	restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=password
    depends_on:
      - prometheus
      - cadvisor

I have Jenkins running on my Docker host at port 8080 already, so to avoid a conflict I’ve mapped host port 8181 to container port 8080 for the cAdvisor container

Deploy Infrastructure

• scp -rv <monitoring-folder-location> mjrod@<docker-host-ip>:/tmp
• mv /tmp/monitoring ~/ on the Docker host to move monitoring folder from temp folder to user directory
• cd monitoring to move into the required directory
• docker compose up -d to start the container services

Containers running successfully

• Navigate to <docker-host-ip>:9090, navigate to status > targets and check that everything is up and running

All endpoints being captured in Prometheus

Configure Grafana

• Navigate to <docker-host-ip>:3000 to open the Grafana administration panel
• Configuration > Data Sources > Add Data Source and select Prometheus as a data source type
• Rename the data source
• Add http://<docker-host-ip>:9090 as URL
• Click Save & test to see if connection is successful

I searched the Grafana dashboard site for dashboard templates and used the import function in Grafana to set up the following dashboards:

• Node Exporter Full
• Docker Container

Docker container Grafana dashboard with 48 hours of data

Next Steps

1. Add additional endpoints to Prometheus so that I am able to monitor more of my homelab’s infrastructure: adding in my AWS blog instance, TrueNAS deployment, Docker-Swarm cluster, and perhaps some of my personal devices if possible.
2. Build more containers, and take note of the additional strain placed on host resources so that I can accurately do some capacity planning for migrating my AWS resources to a container-based deployment.