Log monitoring with Grafana, Loki & Promtail

Effective monitoring extends beyond just tracking metrics; it involves ensuring the health and functionality of your applications. Metrics alone can’t always reveal errors or issues within your systems. By centralizing your logging, you can send log files to Loki and use Grafana for visualization and searching, providing a comprehensive view of your application’s and potential problems.

Grafana

Grafana open source software enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored. Grafana provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations.

Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of pull.

Promtail

Promtail is an agent which ships the contents of local logs to a Grafana Loki instance. It is usually deployed to every machine that runs applications which need to be monitored.

Setup Loki

We need to create a folder to hold the docker-compose.yml and the config.yml .

mkdir loki

In the folders we will make a docker-compose.yml to create the docker container

nano loki/docker-compose.yml

services:
  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    restart: unless-stopped
    environment:
      - TZ=Europe/Amsterdam # Change this to your timezone
    expose:
      - 3100
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki:/tmp
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - backend

networks:
  backend:
    name: backend

volumes:
    loki:
      name: loki

For Loki to run we need a loki-config.yaml for the configuration

nano loki/loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory  # Consider using 'consul' or 'etcd' for production

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

querier:
  max_concurrent: 500  # Adjust based on CPU and memory

query_scheduler:
  max_outstanding_requests_per_tenant: 1000  # Adjust based on load

frontend:
  max_outstanding_per_tenant: 2000  # Adjust based on load

limits_config:
  max_global_streams_per_user: 5000  # Adjust based on actual usage
  ingestion_rate_mb: 50  # Adjust based on actual load
  per_stream_rate_limit: 50MB  # Adjust based on actual load

Setup Promtail

We need to create a folder to hold the docker-compose.yml and the config.yml .

mkdir promtail

In the folders we will make a docker-compose.yml to create the docker container

nano promtail/docker-compose.yml

services:
  promtail:
    image: grafana/promtail:2.9.2
    container_name: promtail
    restart: unless-stopped
    environment:
      - TZ=Europe/Amsterdam # Change this to your timezone
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml:ro
      - /var/log/:/logs # location of the log files to import
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - backend

networks:
  backend:
    name: backend

For Promtail to run we need a promtail-config.yaml for the configuration

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: authlog
  static_configs:
  - targets:
      - authlog
    labels:
      job: authlog
      __path__: /logs/auth.log

- job_name: syslog
  static_configs:
  - targets:
      - syslog
    labels:
      job: syslog
      __path__: /logs/syslog

This config will grab the systems auth & syslog logging.

In scrape_configs change the job_name, targets, job and __path__ to your liking and needs.

Now we will start Loki and Promtail by running:

docker compose -f loki/docker-compose.yml up -d
docker compose -f promtail/docker-compose.yml up -d

Grafana Datasource

We will need to add Loki as a datasource to Grafana.

1. Click Connections in the left-side menu.
2. Search for Loki
3. Click Add new Datasource
4. Enter the name loki
5. Fill in the URL http://loki:3100

Now Promtail will grab the log files, sent them to Loki, which will offer them to Grafana as a datasource to display them. To see the logging:

• Open Grafana
• Click in the menu on Explore
• Choose Loki as your datasource
• Select the label filename and /logs/syslog as value