ElastAlert —Configurations & Frequency Rule Type to Email — Day 2

Before Diving into Frequency rule type , let see some configuration common to rule types.

ElastAlert has global configuration file “config.yaml” which defines several aspects of its operation.

Configurations:

1. es_host : host name of elasticsearch cluster. elastalert records metadata about it searches in es_host (which seen in previous elastalert index stores state). when ever elastalert starting “it will query about the time that it was last run”, this way even elastalert is stopped and restarted we will never miss data.Can be Overridden to each rule by ES_HOST.
2. es_port : port corresponding to es_host. Can be Overridden to each rule by ES_PORT.
3. es_username (optional) : basic-auth username to connect es_host.Overridden to each rule by ES_USERNAME.
4. es_password (optional) : basic-auth password to connect es_host.Overridden to each rule by ES_PASSWORD.
5. es_url_prefix (optional) : URL Prefix for elastic search endpoint.Overridden to each rule by ES_URL_PREFIX.
6. es_send_get_body_as (optional) : Method for querying elastic search (GET,POST or source) Default GET.
7. es_conn_timeout (optional) : sets timeout for connecting to and reading from es_host. Default value is 20.
8. buffer_time : elastalert continuously query against a window from present to buffer_time ago.In this way logs can be filled up to certain extent and elastalert process the events.This Option is ignored for rules when use_count_query or use_terms_query set to true.
9. use_ssl (optional) : Whether or not to connect es_host using TLS (set true or false). Can be Overridden to each rule by ES_USE_SSL.
10. verify_certs (optional) : Whether or not to verify TLS certificates (set true or false). default is true.
11. client_cert (optional) : path to PEM certificate to use client certificate.
12. client_key (optional) : path to private key file to use client key.
13. ca_certs (optional) : path to CA cert bundle to use to verify SSL Connections.
14. rules_folder : Name of the folder contains rule configuration files.Elastalert will load all files in this folder and sub-directories that ends in .yaml format.If the content of this folder is changed, elastalert will load,reload or remove the rules based on respective config files.
15. scan_subdirectories (optional) : sets whether or not elastalert should recursively descend the rules directory — true or false.Default is true.
16. run_every : How often elastalert should query elasticsearch.Elastalert will remember the last time it ran the query for given rule.“Periodically query from that time until the present”. Format of field is a nested unit of time. “minutes : 5”
17. writeback_index : index on es_host to use.
18. max_query_size : Maximum Number of documents will be downloaded from elasticsearch in a single query.Default is 10000.If we expect to get near this number we can use use_count_query for the rule.If limit is reached elastalert will scroll through the pages the size of max_query_size until processing results.
19. scroll_keepalive: The maximum time the scrolling context should be kept alive.(Formatted in Time Units). “Avoid using high values as it abuses in elastic search but to be mindful to allow sufficient time to finish the processing all the results).
20. max_aggregation: Maximum number of alerts to aggregate together,If a rule has aggregation set,all the alerts within time frame will be sent together. Default is 10,000.
21. old_query_limit : Maximum time between the queries for elastalert to start at the most recently run query. When Elastalert starts ,for each rule it will search elastalert_metadata for the most recently run query and start from that time unless its older than old_query_limit in which case it will start from present time.Default is One Week.
22. disable_rules_on_error : ElastAlert will disable rule when rule throw uncaught (not EAException). It will upload trackback to elastalert_metadata & if notify_email is set then send an email notification stating Rule will be no longer run until either elastalert restarts or rule file has been modified. Default is true.
23. notify_email : an email address or a list of email address to which notification emails will be sent.The from address, SMTP host, and reply-to header can be set using from_addr, smtp_host ,email_reply_to options, respectively. By default, no emails will be sent.
24. from_addr : Address to use as from header in email notifications. This from_addr will be for email alerts as well, unless rule over written in rule config. Default is “ElastAlert”.
25. smtp_host : SMTP Host used to send email notifications.This smtp_host will be for email alerts as well, unless rule over written in rule config. Default is “localhost”.
26. email_reply_to : sets the reply to header in emails .Default is recipient address.
27. aws_region: makes elastalert to sign HTTP requests when using Amazon Elastic search service.It will use instance role_keys to sign the requests.Can be overridden by AWS_DEFAULT_REGION.
28. boto_profile : (Deprecated) to use when using Amazon Elastic search service if you don’t want to use instance role keys.
29. profile : AWS profile to use when using Amazon Elastic search service if you don’t want to use instance role keys.can be overridden by AWS_DEFAULT_PROFILE.
30. replace_dots_in_filed_names : ElastAlert will replace dots in filed names with an underscore before writing documents to elasticsearch.Default False.
31. string_multi_field_names : If set , the suffix to use for sub field for string multi-field in elasticsearch.Default value for elasticsearch 2.0 is .raw and .keyword for elasticsearch 5.0
32. add_metadata_alert : If set (true) alerts will include metadata described in rules.(Category,description,Owner and priority), Default is False.
33. skip_invalid : If set true skip invalid files instead of exiting.
34. By default elastalert will use simple basic logging configuration to print log messages to standard error we can change log level to INFO message by using — verbose or — debug.

What is frequency in elastalert ? — When ever Match X in Y times we called it as frequency.

How to define frequency rule file ?

#elastic config this overridden from global config 
es_host: 127.0.0.1
es_port: 9200
#name of rule 
name: Air Search Failure rule 
#rule type <- frequency
type: frequency
#index <- airinfo in elasticsearch
index: airinfo*
#default is_enabled will true to load all rules, but i have #specified this rule to enable
is_enabled: true
#load 5 minutes data from present time in airinfo index for every 1 #minute with minimum events occurred 1 
buffer_time:
  minutes: 5
run_every:
  minutes: 1

num_events: 1

timeframe:
  minutes: 1
#query to fetch from airinfo rule by matching term search and query #string sourceName:SPICEJET
filter:
- term:
    analyticstype: SEARCH
- query:
    query_string:
      query: "sourceName:SPICEJET"
#subject to email
alert_subject: "Search Spike"
#email config to send alerts
alert:
- "email"
email:
- "prabhu@technogramsolutions.com"

How to test elastalert rules ? —

from terminal $elastalert-test-rule —config configfile.yaml — rule rule_folder/rulefile.yaml

EX : elastalert-test-rule — config config.yaml alert_rules/airinfo/search-failure/search-air-1day-digest.yaml

You can see some stats like :

How to run elastalert rules ? — from elastalert directory $elastalert

That’s enough you can see rules loaded and running.

That’s it how frequency rule running.

Hope the basic things are clear,in next part let see about different rule type in depth.

In case of queries please feel free to comment!.

Thanks for the time.