bluekiri
Published in

bluekiri

Kibana mail alerting with Logstash

Intro

A lot of discussion has been going on about how it is possible that Opendistro alerting plugin for Kibana and Elasticsearch doesn’t include an email output for active triggers (https://github.com/opendistro-for-elasticsearch/alerting/issues/3)

In Bluekiri, our Architecture and Sysops teams have quickly developed an alternative design which can be easily implemented with just the addition of another service: Logstash.

Design

The design is based in the usage of two Logstash plugins: the HTTP input plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html) and the email output plugin https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html).

You can have a peep at both guides to see their functioning and example usages, but simplifying things, the HTTP plugin opens a port where you can receive HTTP requests and the mail plugin sends and email when an output is received.

Simple example

A basic Logstash configuration would consist in:

  • input-http.conf (opens 8080 port)
input { http { } }
  • output-mail.conf (sets the destination and structure of the mail)
output {
email {
to => '%{destination_mail}'
address => 'mail server hostname'
subject => 'Alert - %{trigger}'
body => "Trigger: %{trigger}\n\nMonitor: %{monitor}\n\nSeverity: %{severity}\n\nDescription: %{description}"
port => 25
}
}
...

In Bluekiri, we also add a filter to change the mail destination depending on the trigger name:

  • filter-mail.conf
filter {
if [trigger] =~ ".*SYSOPS.*" {
mutate {
add_field => { "destination_mail" => "sysops@mail.com" }
}
}
else if [trigger] =~ ".*ARCHITECTURE.*" {
mutate {
add_field => { "destination_mail" => "architecture@mail.com" }
}
}
...

Once we have the Logstash service configured, we add it as a custom webhook in the Kibana Opendistro alerting destinations:

Now you only need to create your monitors and triggers to get you alerts.

Hope you can find it helpful and apply this design!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store