Getting started with Elasticsearch with a practical example

A simple alerting system that is powered by Elasticsearch.

The overall architecture of the Alerting system

If you are not familiar with the ELK stack, you might want to read this article in detail.

As an engineer, I have implemented a lot of personal and work projects which have used Elasticsearch. However, none is as simple and important as the simple alerting system.

We need to install Elasticsearch, Logstash and Kibana for the alerting system to work. More details are present here. Please make sure that you install all the components with version 2.x .

Step 1. Get your data ready

There are various ways you can send data to Elasticsearch. The simplest by far is sending your log files to Elasticsearch via Logstash ( the letter L in the ELK stack).

In order to generate that data I have used the following log4j config file. This will create a file called “logging.log”

The file can be generated by the two snippets of code alternatively in your Java Class file inside the main method:

log.info(",www.google.com,200,PASS);
log.info(",www.google.com,502,PASS);

Also the logging.log file will contain lines that are similar to the below lines:

2017-02-09;17:08:12:575+0530,www.google.com,200,PASS
2017-02-09;17:08:12:599+0530,www.google.com,200,PASS
2017-02-09;17:08:13:558+0530,www.google.com,502,FAIL
2017-02-09;17:08:13:833+0530,www.google.com,200,PASS

We need to run this program for a while so that we have minimum 10 K or 1 million entries.

2. Logstash config file

The logstash config file will be as below:

You need to change XXXXXXXXX as per your requirements. Please take a note of comments in the above file.

The important things are:

1. The path of log files
2. Username and password values (Gmail account)
3. The email address in TO, CC
4. The timezone should follow the joda time format.

Email messages will be sent to the email address mentioned in “TO” and “CC” address; as soon as Logstash reads the value “FAIL” in the log file.

We can now pass this config file to Logstash via the following command:

bin/logstash -f logstash.conf

3. Visualizing in Kibana

If everything goes well, you can see your data in Kibana as below:

KIbana displaying data

Kibana is available at port 5601 by default.

You can see the failures too by entering Result:FAIL, in the text box.