Monitor Traefik with Grafana, Prometheus & Loki

Secretly we all like pretty graphs with lots of data. Especially when it comes to services we host. Luckily Traefik can expose metrics about the EntryPoints, Routers and Service etc. We will use Prometheus to scrape the metrics and Promtail with Loki to grab the log files.

Setting up Grafana, Prometheus, Promtail and Loki is out of scope of this Story. See my other stories on how to setup Grafana & Prometheus and Promtail & Loki

Metrics

Traefik

To have Traefik expose the metrics for Promethues will need to adjust the configuration for both applications.

In the Traefik configuration we need to add the metrics sections of the settings.

Add the below to your existing traefik.yml

metrics:
  prometheus:
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true

By default Traefik will use the EntryPoint traefik of the dashboard to expose the metrics. Also when the dashboard is disabled. It uses :8080/metrics

Prometheus

In the Prometheus configuration we need to add the scrape config. We will define the job_name , scrape_interval and the target

Add the below to your existing prometheus.yml

# scrape_configs: #Optional when its the first scrape job
  - job_name: 'traefik'
    scrape_interval: 5s
    static_configs:
      - targets: ['traefik:8080']

Note: If your Prometheus and Traefik are not on the same docker network you need to fill in the correct IP.

Now Prometheus will scrape the metrics every 5 seconds. You can adjust this to your liking.

To apply all the configuration changes we made we need to restart Traefik and Prometheus

docker restart traefik prometheus

Log files

Now we have the metrics being scraped byPrometheus, we also want the logging to be montiored.

Traefik

Traefik has two types of logging:

Traefik logs: everything that happens to Traefik itself (startup, configuration, events, shutdown, and so on).

Access logs: everything about the requests being made with information like Entrypoint, HTTP codes, routers, IP Address etc.
Example:

{"ClientHost":"xxx.xxx.xxx.xxx","ClientPort":"40610","DownstreamContentSize":0,"DownstreamStatus":204,"Duration":4791954,"OriginContentSize":0,"OriginDuration":4609995,"OriginStatus":204,"Overhead":181959,"RequestContentSize":400,"RequestCount":78,"RequestHost":"site.yourdomian.com","RequestMethod":"POST","RequestPath":"/yoursite","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"nginx@docker","ServiceAddr":"172.22.0.5:8080","ServiceName":"nginx@docker","ServiceURL":"http://172.22.0.5:8080","StartLocal":"2024-06-02T20:51:17.219573695+02:00","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-06-02T20:51:17+02:00"}In the traefik.yml Add the following code

In the Traefik configuration we need to add the accesslog and log sections at the bottom of the file.

Add the below to your existing traefik.yml

accessLog:
  filePath: "/log/access.log"
  format: json
  fields:
    defaultMode: keep
    names:
      StartUTC: drop

log:
  filePath: "/log/traefik.log"
  format: json

Traefik will timestamp each log line in UTC time by default. To make sure the logs will be with our local timezone we drop the StartUTC.

You can add more field to be dropped if you don't need them. All the available field can be found here.

We will need add a volume to the Traefik docker-compose.yml to mount the right directory in the container.

    volumes:
      - /var/log/treafik:/log

To apply all the configuration changes we made we need to re-create the Traefik container. Otherwise the new volume wont be added to the container.

docker compose up -d --force-recreate

Promtail

In the Promtail configuration we need to add the scrape config. We will define the job_name , __path__ and the target .

Add the below to your existing promtail-config.yaml

scrape_configs:
- job_name: treafik
  static_configs:
  - targets:
      - treafik
    labels:
      job: treafik
      __path__: /logs/treafik/*log

To apply all the configuration changes we made we need to restart Promtail.

docker restart promtail

Grafana Dashboard

Now it's time for the fun part: Building the Dashboard!
As there are multiple ways to build dashboard depending on what you want to see. You can find a overview of all the metrics here. You can make your own dashboards or use mine as a starter.

• Traefik Dashboard

Congrats you have now setup monitoring for Traefik with Grafana, Prometheus, Promtail, Loki!