An Introduction to ElastAlert & Set Up

Simple Framework alerting mechanism

Simple Framework for Alerting anomalies,spikes and other patterns from data in elasticsearch.

We use ElasticSearch,Log stash,Kibana for managing over increasing amount of data and logs, where kibana is great visualizing & querying data, but it need companion tool for alerting when inconsistencies in our data — where Elastalert is created.

Overview — Reliable , Modular ,Ease Set Up & Configure.

Why its Reliable? — in event of restart elastic search or in case of unavailable elasticsearch.

1. ElastAlert— save its state to elasticsearch (when it started & resume previously stopped)
2. If Elasticsearch is unresponsive,ElastAlert wait until it recovers before continuing.
3. ElastAlert (alerts) which throws Errors may be automatically retried for Period of time.

Why its Modular? — Main due to 3 Components.

1. Rule Types : Responsible for Processing Data returned from elasticsearch(basically by initializing the rule configuration & parse data -return data by querying elasticsearch).
2. Alerts : Responsible for Taking action based on match.
3. Configuration : Global Configuration file (config.yaml) — defines several aspects of its operation.(Ex: es_host,es_port,type…)

How it works? — by combining elasticsearch with two types of components, rule types and alerts.

Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match.

This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts.

Several Rule types with common monitoring are included with ElastAlert:

1. Match when there are X events in Y time — Frequency
2. Match when the rate of events increase/decrease — Spike
3. Match when there are less than X events in Y time — flatline
4. Match when there a certain field matches a blacklist/whitelist — blacklist/whitelist
5. Match on any event matching a given a filter — any
6. Match when a field has two different values within some time — Change

Support In Built are — Jira ,email, Google Chat,theHive,Telegram,Slack…

In Addition to this basic usage elastalert supports :

1. Alert links to Kibana Dashboard
2. Aggregate Counts for arbitrary fields
3. Combine alerts into periodic reports
4. Separate Alerts by a unique field
5. Intercept/enhance match data.

Installation :

• Clone Repo : git clone https://github.com/Yelp/elastalert.git or pip install elastalert
• See Requirements in Clonned Repo and install using pip.
• Elasticsearch
• Python3
• python setup.py install

Yes! we have done installation for ElastAlert, let see configuring elasticsearch to elastalert and configuring built in type queries in next part.

In case of queries please feel free to comment if any issues!.

Thanks for the time.