Monitoring multiple Kubernetes clusters in IoT

Authors: Alexandre Gilles-Lordet, Florent Martin

In the e-Health domain, use cases such as fall detection, patient monitoring are critical and must rely on highly available infrastructure and reliable monitoring capabilities.

In a previous article, we introduced ELCA’s IoT e-Health platform. This platform has IoT devices connected to edge Gateways running k3s (a lightweight Kubernetes implementation). Gateways are forwarding sensor data to a server for storage, processing and serving data to downstream consumers. The server also runs a Kubernetes cluster.

In this article, we will explain how we monitor the gateways, to detect incidents or failures and notify administrators. Our implementation is based on InfluxDB and Telegraf.

Photo by Chris Liverani on Unsplash

As shown in the figure below, the device is connected to the gateway by the interfacing module (1), which translates data into MQTT messages respecting the InfluxDB Line Format. Formatted data are sent to the gateway broker (2). The messages can go through some data processing by using specific MQTT topics (3 and 4) and are forwarded to the server broker through an MQTT bridge (5). The server broker sends the messages to a Telegraf instance via an MQTT input plugin (6) who then push the InfluxDB time-series database (7).

Functional architecture of the platform (partial)

Our operational goal is to obtain gateway metrics like the CPU or memory usage, or the state of the Kubernetes pods on the gateway cluster. The leading open-source solution for monitoring is Prometheus, and it is widely used with Kubernetes. However, its pull-based operation principle is not well suited to our case since we are not keeping an updated central list of all the gateways in the system and how to reach them. In consequence, push-based monitoring is favoured, so we selected the Telegraf — InfluxDB solution.

Telegraf for monitoring

Telegraf has a lot of plugins, among which plugins to fetch the CPU and memory usage and the state of Kubernetes clusters. We thus deploy a Telegraf instance on each of our gateways using Fleet as the central deployment tool. We applied the following input configuration:

[[inputs.kubernetes]]
     url = "https://$HOSTIP:10250"
     bearer_token =       "/run/secrets/kubernetes.io/serviceaccount/token"
     insecure_skip_verify = true
     fieldpass = ["cpu_usage_nanocores", "memory_available_bytes", "memory_usage_bytes", "fs_available_bytes", "fs_used_bytes"]
     namepass = ["kubernetes_node", "kubernetes_pod_container"]
     tagexclude = ["node_name"]
[[inputs.cpu]]
     percpu = true
     totalcpu = false
     collect_cpu_time = false
     report_active = false
     fieldpass = ["usage_user", "usage_system", "usage_idle"]
[[inputs.mem]]
     fieldpass = ["used", "total"]

Notice that we filter out a lot of fields we are not interested in with “fieldpass”, to limit the performance impact of the high number of metrics produced by default by these input plugins.

There is also a specific configuration to allow the Telegraf pod to connect to the Kubernetes API, so it is recommended to use this helm chart, or get inspired by it.

MQTT everywhere

We want to store the relevant metrics in a new InfluxDB database. They are separated from the sensor metrics for security and privacy of the medical data, and to allow for easy access policies as the persons with access to the medical data are different from the ones who will monitor the platform.

From this point, we could just open the API of the monitoring database and tell the monitoring Telegraf instances to store their metrics directly through this API, however, it is hard to secure (we need to deploy InfluxDB tokens on each gateway and update our provisioning process) and it does not ensure that the MQTT bridge is working correctly with medical data flowing through.

To solve this security challenge, we decided to use the same technique as the one used for the sensors’ data. It means that the gateway Telegraf instances are configured to publish to a specific topic, as depicted by the output configuration below:

¬[[outputs.mqtt]]
     servers = ["gateway-broker:1883"]
     topic_prefix = "toServer/monitoring/"
     qos = 2

As the messages reach the server, they are forwarded by the server broker to a new monitoring Telegraf instance with a configuration similar to the one used for medical data:

¬[[outputs.influxdb_v2]]
     urls = ["http://influxdb2-monitoring"]
     organization = "influxdata"
     bucket = "default"
     token = "${INFLUX_TOKEN}"
[[inputs.mqtt_consumer]]
     servers = ["tcp://server-broker.data-in:1883"]
     topics = [
          "fromGateway/monitoring/#",
     ]
     qos = 2
     persistent_session = true
     client_id = "telegraf-monitoring"
     data_format = "influx"

This Telegraf instance will store the monitoring data in our new InfluxDB database.

The figure below depicts the full architecture to transmit the metrics:

Monitoring architecture

Monitoring dashboards and alerts

InfluxDB comes with built-in dashboards and alerts. We created a simple dashboard showing the list of the gateways that connected to the server in the last 30 days, and their status, e.g. down or up and running. We can select a specific gateway and see its CPU, memory and filesystem usage.

Gateway monitoring dashboard in InfluxDB

The query to display the gateway table is the following:

¬import "influxdata/influxdb/monitor"
import "experimental"
 
from(bucket: "default")
  |> range(start: -30d)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["_field"] == "usage_idle")
  |> filter(fn: (r) => r["cpu"] == "cpu0")
  |> group(columns: ["host"])
  |> monitor.deadman(t: experimental.subDuration(d: 1m, from: now() ))
  |> group()

The critical part is the “monitor.deadman” function, which returns true if the gateway has not emitted CPU monitoring data for one minute from now. It is paired with the table cell type where we removed all the columns but the gateway hostname and the one created by “monitor.deadman”.

For alerts, we configure deadman checks through the UI, which is pretty straightforward.

Conclusion

Thanks to the capabilities of InfluxData’s Telegraf, we could quickly retrieve monitoring metrics on our gateways and store them on the server. Note that similarly, we can add logging features with the tail input plugin.

Based on load tests, we outlined that Mosquitto MQTT brokers are far from being the throttling points under a high load (Telegraf and InfluxDB are struggling first). We thus designed an architecture based on the same MQTT bridge or both monitoring and device data.

Acknowledgments

This article is based on the work of ELCA IoT team with the contributions of Alexandre Gilles-Lordet, Antoine Hue, Paul Bernet, Sébastien Morand, Daniel Raimundo, Sonia Duc and Marco Maccio.