Icinga Notifications in Telegram

Yannik Schmidt
Jun 29, 2020 · 5 min read

There are multiple solutions for mobile notifications from Icinga2, including native apps or via E-Mail. However I found that problems on my server have actually become rare and I don’t check my private mails as often as I used to. So I thought it would be great to just get my alerts via a communication channel I use all the time anyway and which supports push notifications: The messenger Telegram.

Telegram has a feature called bots which is more or less a fancy interface to send chat messages to clients. You create a bot from your Telegram-client and receive an authentication token which you can then use to run a handler on your server. After this, whenever the bot (which is running on the Telegram servers) receives a message, it gets dispatched (by the telegram servers) to the handler on your server and you can process it there.

In theory the bot-handler doesn’t necessarily need to run all the time, only for subscribe or unsubscribing in our case, so in theory we could fire up a temporary handler, subscribe to the bot in our Telegram client and then only ever run a notification script from Icinga that directly talks to the Telegram bot API.

However I decided for reasons of simplicity and potential extensibility to create a simple HTTP Gateway that will forward an incoming request via HTTP (which we can much more easily create in an Icinga notification script) and translate this to the API. That has the added benefit of letting the gateway handle new subscribe requests at any time.

The official Telegram Documentation describes the setup of a new bot in detail, the short form is: Add a contact called @BotFather, press ‘Start’ and follow the instructions. You will be given an authentication token which we will use later.

You can find my Telegram Gateway on GitHub. If you have a look at the file telegram-interface.py, you can see it functions very simple:

  • start a Telegram bot (using the python-telegram-bot library)
  • add a handler that saves the chat-id of anybody writing /start to the bot
  • add a handler that allows people to unsubscribe via /unsubscribe (removing the respective chat-id)
  • start a flask server listening locally for POST-Requests with a special JSON-payload
  • if a request arrives, format a message as Markdown and write it to all chat-id’s (respectively all subscribed clients) in the save file

To start the gateway:

git clone https://github.com/FAUSheppy/telegram-http-gateway
cd telegram-http-gateway echo $AUTH_TOKEN > bot.token
python3 -m pip install -r req.txt
./telegram-interface.py --port 6000

With $AUTH_TOKEN being the token you received when creating the bot. You can of course use any port, 6000 is port use in the later examples.

The following part assumes you have a working Icinga2-instance. Icingaweb is not needed. On most systems just installing Icinga2 via the package-manager will create a working instance ( apt install icinga2 on Debian). If you don't care about the explanation you can just clone the atlantishq monitoring-tools into /etc/icinga2/monitoring-tools/ and add an

include /etc/icinga2/monitoring-tools/commands.d/telegram-notify.conf;

at the end of icinga.conf. Make sure the directory and it’s contents can be read by Icinga.

Notification Command Object

Then we need the script which Icinga actually executes to create a request to the gateway. In theory this could just be inline-curl in a NotificationCommand-object:

object NotificationCommand "telegram-service-notification" {
command = [ "/usr/bin/curl", "-X", "POST", "-H", "Content-Type:
application/json", "--data", "
{ 'service_name' : 'hello world',
service_type : '$notification_type$' }",

But that gets very messy, very quickly so I created a small python-script that emulates the arguments of the mail notification script already in included in Icinga, so you can just copy and paste the NotificationCommand for mail notifications (usually in notifications.conf in /etc/icinga2) and replace the path/name. It should look like this afterwards:

object NotificationCommand "telegram-service-notification" {
command = [ "/etc/icinga2/monitoring-tools/telegram-notify.py" ]

arguments += {
"-4" = "$notification_address$"
"-6" = "$notification_address6$"
"-b" = "$notification_author$"
"-c" = "$notification_comment$"
"-d" = {
required = true
value = "$notification_date$"
"-e" = {
required = true
value = "$notification_servicename$"
"-f" = {
value = "$notification_from$"
description = "Set from address."
"-i" = "$notification_icingaweb2url$"
"-l" = {
required = true
value = "$notification_hostname$"
"-n" = {
required = true
value = "$notification_hostdisplayname$"
"-o" = {
required = true
value = "$notification_serviceoutput$"
"-r" = {
required = true
value = "$notification_useremail$"
"-s" = {
required = true
value = "$notification_servicestate$"
"-t" = {
required = true
value = "$notification_type$"
"-u" = {
required = true
value = "$notification_servicedisplayname$"
"-v" = "$notification_logtosyslog$"

vars += {
notification_address = "$address$"
notification_address6 = "$address6$"
notification_author = "$notification.author$"
notification_comment = "$notification.comment$"
notification_type = "$notification.type$"
notification_date = "$icinga.long_date_time$"
notification_hostname = "$host.name$"
notification_hostdisplayname = "$host.display_name$"
notification_servicename = "$service.name$"
notification_serviceoutput = "$service.output$"
notification_servicestate = "$service.state$"
notification_useremail = "$user.email$"
notification_servicedisplayname = "$service.display_name$"

Notification Object

We then need to apply this notification to something:

apply Notification "telegram-generic" to Service {
import "mail-service-notification"
user_groups = ["icingaadmins"]
command = "telegram-service-notification"
assign where host.address

By convention we import the mail-service-notification object as a basis, then we define user_groups for which we want send notification. This (in our case) is not about whom to send notifications, but for which services we send notifications. Usually you just want the icingadmins group here because this group already exists by default and will get notifications for anything (unless you already changed this behavior). If you have a special notification group with additional filters, rules, etc, you can use this group instead (or use a single user with the “user = … “-directive).

Finally “ assign where host.address” is basically an “ assign everywhere” since every service likely has an address. Obviously you could use something more specific here, like assign where host.vars.telegram_notifications. Please refer to the Icinga2 Documentation for information about the exact workings of the assign-directive.

Now you can add your Bot as a contact in your telegram client, send a /start-message and you will now receive notification from your Icinga in Telegram.

Originally published at https://blog.atlantishq.de.

Anti Clickbait Coalition

Read about the things you want to read about, not the things you got baited into.

Anti Clickbait Coalition

The Anti Clickbait Coalition features many topics, but has strict requirements for story-titles and content, so readers can focus on the articles they really want to read.

Yannik Schmidt

Written by

Python programmer at heart, web developer on my bad days.

Anti Clickbait Coalition

The Anti Clickbait Coalition features many topics, but has strict requirements for story-titles and content, so readers can focus on the articles they really want to read.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store