Historical monitoring with Telegraf, InfluxDB and Grafana

5 min readApr 5, 2024

In my home automation setup I already have live monitoring. It provides me an overview of health of the Linux box, including CPU and memory usage, as well as temperature of the box. While its important to know if current utilization is within a reawonable range, live dashboards don’t provide an information of what happen overnight, or any other time when I was away from Grafana dashboard. It is possible to add a time series chart to my current setup, but it will only show historical data for the time, when dashboard was open.

Adding persistent storage is the next logical step to keep history of the utilization data. In my setupTelegraf sends metrics in InluxDB line format, the easiest way to add persistence is to plug InfluxDB between Telegraf and Grafana.

InfluxDB is available as docker container. I just need to find a LInux box with sufficient amount of storage, and install the container. I choose to have InfluxDB database located in a separate docker volume. To simplify creation of volumes and container, I wrote a docker-compose.yaml:

Started the container with docker-compose up -d, then opened InluxDB GUI on port 8086 (http://ipaddress:8086)

Setup initial user credentials, organization name and bucket name.

Then click “Continue”. The first screen presents administrative token, I copied it to a safe place:

The pressed “Quick Start” and get home screen:

First , lets get API token, so Telegraf can push the data. Click “Upload” icon, then choose “API Tokens”:

Then click “Generate API token”:

As a good security practice, the token should be allow as little as required to push metrics to the database. I allowed writing to the bucket “kvm”, and allowed all Telegrafs.

Click “Generate”:

Copy the token, then close the popup and verify, that the API key is in the list:

Add the output plugin to the telegraf.conf:

Then restart Telegraf: systemctl restart telegraf, and check if any data is coming to InluxDB. Navigate to “Data explorer”:

Seelct “kvm” bucket, then open drop-down for the list of keys, and check if “cpu” is there.

Thats it for InluxDB. It convenient, that it derive structure of bucket from the Telegraf payloads, so no effort required to create a structure or adjust it when source format changes.

Now lets attach Grafana to the InfluxDB. Navigate to Grafana web interface and open “Data sources”. Click “Add new datasource”:

Select “InfluxDB”:

I named this datasource “influxdb”. I choose “InfluxQL” query language, that is most recent out of stable query interfaces to InfluxDB as of April 2024. In the URL field I entered the same URL as I use for InfluxDB http://192.168.8.5:8086. Leave all “Auth” settings disabled.

Scroll further down there are connection settings. “Database” is bucket name. “User” is my user name in the InfluxDB. Password is API token, that I generated in InfluxDB earlier.

Click “Save & test”. With enough luck the following message appear:

Now lets create a new dashboard to display historical data. First, add visualization (chart) for CPU usage. In the “Query” tab I select CPU utilization metrics (from = cpu), field — “usage_idle.

Similarly to live dashboards, I added data transformation to calculate total CPU usage:

Similar to CPU usage chart, I added the remaining charts for memory usage and temperature. Done and saved, here is the historical dashboard:

Now I have two dashboards and two data paths. But since InfluxDB has the most recent metrics, I don’t even need live dashboard. I can add latest gauges to the historical dashboard. Lets start with CPU usage. Duplicate CPU usage time series: