Elastic Stack Familiarization
Phase 01 — Introduction to Elasticsearch — Blog 02
Elastic Stack Familiarization
This is the second part to the “Introduction to elasticsearch” series. This short blog aims to provide a superficial familiarisation of the components in the Elasticsearch stack. What are the uses of these components, how does it provide value to the stack or why they are better options to be used with Elasticsearch.
Components of the stack
A breakdown of the components of the services provided by Elastic, the parent company of Elasticsearch is given in the image below:
1. Data storage,search and visualisation services
1.1 Elasticsearch
We have read about Elasticsearch in detail in the part one of this blog series. The data storage and search part of the Elastic stack is taken care by Elasticsearch. The data storage solution is NoSql where as the search part is handled via customisable and flexible APIs to the powerful library of Apache Lucene.
The documentation for Elasticsearch can be found here .
Another important area concerning to the Elasticsearch is the plugins that are available to be used with it. These plugins mostly are intended to boost a specific functionality or make the usage of Elasticsearch easy in certain scenarios. We will be delving deep into the elasticsearch plugins in a later blog series.
1.2 Kibana
Kibana is an out of the box visualisation tool for the data stored in Elasticsearch. It right away connects to the Elasticsearch database and lists out the database names (indices in Elasticsearch world) for us to start the visualisation on the data in the index/indices. Kibana provide supports for most of the Elasticsearch operations and supplements us with a plethora of visualisation options to visualise the data in the way we want. These visualisations can be used to create dashboards. An example of Kibana dashboard is given below:
2. Data shipping, parsing and log processing
2.1 Logstash
We often have to collect data from various sources, and have to parse them to make it in uniform format and then push to elasticsearch. Logstash is one such data pipeline which takes in data from wide variety of sources and transform it to the required format and then passing the modified data to Elasticsearch. Logstash can push the data not only to elasticsearch but also to a variety of sinks.
The following diagram explains the logstash data pipeline in a very simple manner.
Apart from the mulitple input and output sources supported by Logstash, the main feature of the Logstash is the data parsing options which it offer. It provides a rich enviornment by providing the Groke rule facility to process the incoming logs.
2.2 Beats platform
The beats platform from Elasticsearch are a group of data shippers which can be installed in the source and the logs/files are shipped to Elasticsearch. They can ship to Logstash if the data requires pre-processing before getting into Elasticsearch.
There are currently 5 members in the Beats platform, they are
2.2.1 Filebeat — ships the logs from the installed system
2.2.2 Metricbeat — ships the metric datasets of the installed system
2.2.3 Winlog beat — shipt the logs from a Windows system
2.2.4 Heartbeat — The uptime monitoring parameters are sent from the installed system
2.2.5 Packetbeat — Ships the network data
The flow of a typical beats installed system is as below:
3. Monitoring and Security services
3.1 X-Pack
X-Pack is a paid extension package for Elasticsearch. It provides a clubbed solution of security, monitoring, alerting and reporting capabilities . X-Pack is available only from Elasticsearch version 5.x. For the older versions, these solutions were implemented separately using different packages.
3.2 APM — Application Process Monitoring
Elastic APM is a newbie to the stack and is at alpha at the time of this blog . As the name suggests, APM is used to monitor a specific application by storing and analysing its logs using Elasticsearch and Kibana.
4. Hosted/Enterprise Solutions and Services
4.1 Elastic-cloud
Elastic cloud provides the Elastic stack as a service , which inturn makes it easer to scale and upgrade to the latest versions.
4.2 Elastic Enterprise
Similar to elastic cloud but this one is more scalable and with hot support, intended for enterprise range solutions
Conclusion
In this article, I have explained the basic functionalities of the components in the Elastic stack. In the next installment to this series, we will learn how to install Elasticsearch locally and do some very basic CRUD operations.
