Monitor and maintain the health of your devices with thin-edge.io release 0.7

For those not aware of thin-edge.io, it is the first open-source and cloud-agnostic edge framework designed for resource-constrained IoT edge devices.

So what does that mean:

• thin-edge.io is purpose built to run on devices with limited compute and memory resources using embedded Linux distributions like Debian, Yocto, etc.
• It is cloud agnostic, so you can connect to any cloud platform using “mappers” which only need small modifications to the specifics of each cloud.
• It provides a full set of capabilities to connect and manage your IoT device out-of-the-box with secure cloud connectivity, robust and extensible device management, device monitoring, and more.

On top of that, it is packaged to be very easy to install and run on your edge device, so you can get started quickly.

If you want to know more about thin-edge.io, please visit the website: https://thin-edge.io.

With release 0.7 of thin-edge.io we have improved the device monitoring capabilities and enhanced the Cumulocity IoT integration. Our device monitoring can now monitor the health of the tedge daemon and restart unresponsive services detected by systemd. For implementations connecting with Cumulocity IoT, thin-edge.io now has optional plug-ins that allow you to remotely manage the configuration and log files.

So, in summary, thin-edge.io release 0.7 gives you:

• Health check for tedge daemons
• Watchdog for thin-edge.io
• Configuration file management plugin for Cumulocity IoT
• Logfile management plugin for Cumulocity IoT

In this blog, I will give you just a high-level overview of the new features, with more details being available in the documentation on the thin-edge.io Github.

Health check for tedge daemons

The health of tedge daemon processes like tedge-mapper and tedge-agent can now be monitored via MQTT in release 0.7.

The daemons expose MQTT health endpoints which you can query to determine if the process is active.

You can use the following topic scheme to listen for health check requests:

tedge/health-check/<tedge-daemon-name>

The daemon will then respond back to the topic:

tedge/health/<tedge-daemon-name>

with the following payload:

{ “status”: “up”, “pid”: <process id of the daemon> }

All daemons will also respond to health checks sent to the common health check endpoint tedge/health-check.

The following endpoints are currently supported by various tedge daemons:

tedge/health/tedge-agent
tedge/health/tedge-mapper-c8y
tedge/health/tedge-mapper-az
tedge/health/tedge-mapper-collectd

All future tedge daemons will also follow the same topic naming scheme convention.

The mosquitto bridge clients connecting thin-edge devices to the respective cloud platforms also report their health status as retained messages to tedge/health/<mosquitto-cloud-bridge> topics. The health check messages published by these clients are just numeric values 1 or 0, indicating active and dead bridge clients respectively.

Here are the health endpoints of currently supported clouds, bridged with mosquitto:

Explicit health check requests via tedge/health-check topics are not supported by these bridge clients. Since the health status messages are sent as retained messages, just subscribing to these health topics is sufficient to get the latest status.

Watchdog for thin-edge.io

An alternative way to check the health of your device is using system, which is a replaceable default component of thin-edge.io. The watchdog capability enables systemd to detect when a service is unresponsive and attempts to restart unresponsive services. To detect if a service is healthy or not, systemd monitors the notifications from that service which are sent at regular intervals.

You can enable the watchdog for one or more thin-edge.io services

• tedge_agent,
• tedge_mapper_c8y
• tedge_mapper_az
• tedge_mapper_collectd

First, you need to enable the watchdog feature in the systemd service file

As an example, the watchdog feature for the tedge-mapper-c8y service is activated. Therefore you need to update the systemd service file as shown below.

[Unit]
Description=tedge-mapper-c8y converts Thin Edge JSON measurements to Cumulocity JSON format.
After=syslog.target network.target mosquitto.service tedge_watchdog.service
[Service]
User=tedge-mapper
ExecStart=/usr/bin/tedge_mapper_c8y
Restart=on-failure
RestartPreventExitStatus=255
WatchdogSec=10

This means that every 10 seconds a notification is sentd to the systemd service.

Then you can start and enable the tedge-watchdog service as follows:

systemctl start tedge-watchdog.service
systemctl enable tedge-watchdog.service

The tedge-watchdog service will be sending health check messages to the monitored services with the configured WatchdogSec interval.

You can check the message exchange between the service and the watchdog by subscribing to tedge/health/# and tedge/health-check/# topics. Then you will see regular health checks within the set intervals.

If the watchdog service did not send the notification to systemd within the set interval, then systemd will kill the existing service process and restarts it. This is also shown in the before mentioned topics.

For more information, have a look at the documentation .on the thin-edge.io Github.

Configuration file management plugin for Cumulocity IoT

Configuration management is a new capability in thin-edge.io release 0.7. It gives you access to the configuration of your device from your cloud IoT platform. It is initially available for Cumulocity IoT and will be extended for other platforms in the future. For Cumulocity IoT the plugin essentially provides the client for the Cumulocity IoT configuration management capability.

To enable configuration management, the first step is to install the c8y_configuration_plugin on your device. In the documentation is a script available which is the easiest way to get this running. There are also alternative methods mentioned there, to get the configuration management plugin working.

After installing, connect your device to the cloud and initialize the plugin with:

sudo c8y_configuration_plugin –init

To manage specific configuration files you need to add the lines below to the /etc/tedge/c8y/c8y-configuration-plugin.toml file in the following format:

files = [
{ path = ‘/etc/tedge/tedge.toml’, type = ‘tedge.toml’},
{ path = ‘/etc/tedge/mosquitto-conf/c8y-bridge.conf’, type = ‘c8y-bridge.conf’ },
{ path = ‘/etc/tedge/mosquitto-conf/tedge-mosquitto.conf’, type = ‘tedge-mosquitto.conf’ },
{ path = ‘/etc/mosquitto/mosquitto.conf’, type = ‘mosquitto.conf’ },
{ path = ‘/etc/tedge/c8y/example.txt’, type = ‘example’, user = ‘tedge’, group = ‘tedge’, mode = 0o444 }
]

• Path: full path to the configuration file.
• Type: unique alias for each file entry which will be used to represent that file in Cumulocity.
• user, group, mode: UNIX file permission settings to be used to create a configuration file.

Then you can start the configuration plugin process and enable it on boot by systemctl

sudo systemctl start c8y-configuration-plugin.service
sudo systemctl enable c8y-configuration-plugin.service

Then everything should be working, when you go to the configuration tab (in device management in Cumulocity IoT). There you can find configuration types, as declared in the plugin configuration file in the earlier step.

example of configuration files in Cumulocity IoT.

All configuration updates are notified over tedge/configuration_change/<config-type> MQTT topic, allowing software components installed on the device or a child device to react to these updates. That is a nice way to check what is happening.

Logfile management plugin for Cumulocity IoT

Release 0.7 of thin-edge.io makes log files accessible to your cloud IoT platform. Like configuration management, it is initially available for Cumulocity IoT and will be extended for other platforms later.

First, you need to install the c8y_log_plugin.

When the device is connected to Cumulocity IoT, the plugin needs to be started and enabled as follows:

sudo systemctl start c8y-log-plugin
sudo systemctl enable c8y-log-plugin

Then in device management a new tab called “Logs” is added.

It is possible to add different log types, therefore you must edit the c8y-log-plugin.toml. In this file, you can specify the type and path of the logs to be retrieved.

For example, if you wish to request thin-edge software logs and Mosquitto logs, an example toml file must contain:

files = [
{ type = “software-management”, path = “/var/log/tedge/agent/software-*” },
{ type = “mosquitto”, path = “/var/log/mosquitto/mosquitto.log” }
]

path needs not be a complete path. It can be a full path to a log file or a glob pattern to support log-rotations. For example the “software-management” type is a glob pattern that groups any file inside /var/log/tedge/agent/ that starts with “software-”.

Software-management log-file

The type key in the toml file is the name of the log file which you will be visible in Cumulocity IoT.

I think these latest additions are great for the project as we heard from many developers that these types of functionalities would add great value. I hope that you as a reader agrees with this.

If you got thoughts, questions or comments or any other ideas, please connect on discord and GitHub.

As this is an open-source project, you can find the code on Github as well and also guidelines to participate in the project.

More blogs can be found here: https://medium.com/thin-edge-io https://www.youtube.com/channel/UC9u2hxmXY-cTqM1-aaDBY-w