Custom Alerting with ELK and ElastAlert

How ELK and ElastAlert helped us setup alerts based on business metrics

ELK (ElasticSearch, Logstash, Kibana) is a very popular way to ingest, store and display data. Now this data can be either your server logs or your application performance metrics (via Elastic APM). However, one of the key drivers to application availability is prompt anomaly detection and alerting.

In some cases; there are business related alerts that need to be implemented which cannot be implemented via commercially available APM tools (Either the feature is not available or it requires additional licenses). However, there is one cool tool out there called ElastAlert that allows you to create your own alerts based on Elastic queries. So all your business metrics stored in a mountain of application logs can be identified and alerted upon using this nifty tool.

Elastalert was built by Yelp in order for them to identify inconsistencies with the data. It can be installed on any machine (as long as it has connectivity to your elastic cluster) and is easy to configure.

A Simple Case Study

For one of the projects that I worked with we had a unique problem. There was an APM solution already present; however we needed to setup some business based alerts based on requests from one of the consumers of our service as it will help them design their application accordingly. Even though we had comprehensive monitoring via the APM; our licenses did not extend towards custom alerting based on application logs.

Since our application logs are present in ElasticSearch and we used Kibana for business transaction SLA violation analysis; we wanted to extend it to include alerts not just based on performance SLA but also based on certain business rules/logic involved.

Our focus then turned towards ElastAlert (https://elastalert.readthedocs.io/en/latest/). Once, we installed it in one of the servers which had connectivity to our elastic stack; we configured it for our business based alerts. One such configuration is as given below:

# From example_rules/example_frequency.yaml
es_host: elasticsearch.example.com
es_port: 14900
name: Example rule
type: frequency
index: logstash-*
num_events: 50
timeframe:
    hours: 4
filter:
- term:
    some_field: "some_value"
alert:
- "email"
email:
- "elastalert@example.com"

So essentially, you provide the elastic host (data node), port. Name can be any meaningful name for the rule. There are several types of Elastalert rules but for our purpose; the frequency rule was sufficient enough. We needed to know certain business events occurred within a given timeframe. Then we specify the index and the query for which will satisfy the business condition to pull the data. Our alerting mechanism was via email; so we provided the email of our alerts inbox. There are a lot of integration points that elastalert provides (Email, Slack, Wiki, JIRA to name a few).

For each new business rule created; prior to restarting the elastalert process; we can test each of the rules created via the command

elastalert-test-rule example_rules/yourrule.yaml

Once we had the alerts up and running; we also noticed that sometimes the process went down (due to routine maintenance on the machine where elastalert is hosted). In order to eliminate process termination; we created a small “elastalert doctor” script which ran as a CRON job every 5 minutes to monitor the ElastAlert process and then bring it up if it has gone down. It can be extended to take a backup of the Elastalert logs in case the process crashed which can be used to analyze what the issue was when the process went down.

#/bin/bash
if ps -aux | grep -v grep | grep elastalert > /dev/null
then
echo "Process is running"
else
nohup python -m elastalert.elastalert --verbose
fi

And viola! now we have Elastalert running and alerting at all times.

Note: There is integration with Kibana available now for ElastAlert. That is a work in progress and I shall write about my experiences with it soon.