Kibana mail alerting with Logstash

Intro

A lot of discussion has been going on about how it is possible that Opendistro alerting plugin for Kibana and Elasticsearch doesn’t include an email output for active triggers (https://github.com/opendistro-for-elasticsearch/alerting/issues/3)

In Bluekiri, our Architecture and Sysops teams have quickly developed an alternative design which can be easily implemented with just the addition of another service: Logstash.

Design

The design is based in the usage of two Logstash plugins: the HTTP input plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html) and the email output plugin https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html).

You can have a peep at both guides to see their functioning and example usages, but simplifying things, the HTTP plugin opens a port where you can receive HTTP requests and the mail plugin sends and email when an output is received.

Simple example

A basic Logstash configuration would consist in:

• input-http.conf (opens 8080 port)

input { http { } }

• output-mail.conf (sets the destination and structure of the mail)

output {
  email {
    to => '%{destination_mail}'
    address => 'mail server hostname'
    subject => 'Alert - %{trigger}'
    body => "Trigger: %{trigger}\n\nMonitor: %{monitor}\n\nSeverity: %{severity}\n\nDescription: %{description}"
    port => 25
  }
}
...

In Bluekiri, we also add a filter to change the mail destination depending on the trigger name:

• filter-mail.conf

filter {
    if [trigger] =~ ".*SYSOPS.*" {
      mutate {
          add_field => { "destination_mail" => "sysops@mail.com" }
      }
    }
    else if [trigger] =~ ".*ARCHITECTURE.*" {
      mutate {
          add_field => { "destination_mail" => "architecture@mail.com" }
      }
    }
...

Once we have the Logstash service configured, we add it as a custom webhook in the Kibana Opendistro alerting destinations:

Now you only need to create your monitors and triggers to get you alerts.

Hope you can find it helpful and apply this design!