An Introduction to Elastic Search

While before jumping to Elastic Search, lets know few about!

What is Elastic Search ? — Open Source,Distributed,Restful Search & Analytics — Full Text Search Engine capable of addressing a huge number of cases.

1. We can build complex search functionalities ( Ex : Auto complete search box or to show appropriate content (synonyms) , highlighting content , correcting typo’s) — Ex : Google Search Box..or Implementing Search Box for Web Shop.
2. Analytics Search Engine — Were we can query and analyze structure data — can make pie charts,lie charts, or what we need! .
3. Not Really a Business Intelligence but we can indeed lot of valuable data that stored in elastic search.
4. Analyze Application logs & System Metrics — logs from application and various system ,errors & CPU memory usage metrics then analyze this perhaps with alerting system.
5. APM (Application Performance Management) — Common Use case in Elastic Stack (will see on upcoming parts)
6. Sending Events to Elastic Search … (for Aggregations,Analysis)
7. Excellent at analyzing data. — Based on historic data help to predicate future.
8. Anomaly Detection — Machine learning (if any deviations from normal behavior help to notified such as email,message)..

How Does Elastic Search Work ? — In Elastic Search data is stored as documents. (similar to row in relational data bases ex.MySQL) and a document’s data is separated into fields (similar to column in relational data bases). — Way we Query Document is by Rest API.

1. Elastic Search is written in java & built on Apache Lucene.
2. Easy to use and highly scalable.
3. Complex Technologies Inside ,simpler to use and used by large companies.

Overview of Elastic Stack— Developed by Elastic Search BV and “Elastic Stack comprises — X-Pack,Kibana ,Logstash,Beats and Elastic Search”

Kibana :

1. An Analytics and Visualization platform — use many build in visualizations, set up machine learning in kibana.
2. Can Manage Parts of elastic search and logstash.
3. Build Dashboards for Admins,developers..,KPI (Key Point Interest)

Sample Kibana Dashboard:

Logstash: — General Purpose tool — Data Processing Pipe Line. A data that receives logstash handled as events which can be anything of choice (calls,log file entries,commerce orders) — Processed by logstash and shipped off to one or more destinations (Kafka queue,email,,elastic search ,to http end points)

Ex: Each States — (Left Side of images) are Input plugin were logstash will read the events and shipped to (Right Side of images) are Output plugin (stashes) and in middle logstash tool Filter Plugins were we can pass csv,xml,json and help us to look up data.

1. So in Nutshell, were logstash will receive events from one or more inputs and send them to one or more stashes(output plugins), logstash in horizontally scalable.
2. Defined in proprietary markup format (similar to json) not only mark up language how ever we can add also conditional statements and make pipeline dynamic.
3. Access logs from any web server or file , we can instruct logstash to read line by line from input login (later in this tools we have beats handy tool), log files contains vary pieces of information ,were logstash receives line as string in other piece we have to make unstructured data to structure data by Grok pattern (like regular expression).

X-Pack : (Part of Elastic Stack) — Add Additional Features to Elastic Search & Kibana.

1. Security — Add Authentication & Authorization to Both , were kibana can integrate with authentication providers and Control permissions with grained authorization — Helpful when different people need different kind access.
2. Monitoring — Gain Insight How elastic search is running (specifically CPU,Disk Space & Memory Usage etc..) and many useful metrics..
3. Alerting — can be used for lot of things (ex : if a web server CPU’s usage exceeds some X%, or if application errors reaches threshold
4. Reporting — can be generated on demand or scheduled- when certain conditions are fulfilled,customize the looks and feel the reports and exports the data as CSV.
5. Machine Learning — (Interface provided by Kibana) — helps to detect Abnormality detection ( deviates from normal behavior) and forecasting future.
6. Graph — Relationship in Data — to show related data — uncommonly common signals relevance not popularity (out looking at popularity is misleading). -provides interface with kibana and integrate into applications api.
7. SQL — Queries written in Query DSL, flexible , but also a bit verbose at first. (Send SQL over HTTP or through a JDBC Driver)

Beats: — Collection of Data Shipper (light weight agents) -lot number of betas available. (File Beat, Metric Beat, Heart Beat..etc)

Alright! Put all pieces together now.

Elastic Search : (Data) — Ingesting data can be done by Logstash or Beats but also directly to elastic search (by elastic search api).

Kibana : User Interface (Runs top of Elastic Search) — Really Powerful tool to build Dashboard and machine learning.

Might Some how we heard about ELK Stack (Elastic Search,Logstash,Kibana), before originating beats, after it Elastic Stack.

From previous we have understand what is elastic stack,now in next part lets get dig more on Architectures and installation.

In case of queries please feel free to comment!.

Thanks for the time.