Elastic Stack: An Introduction

The Intro to a multipart series of using and configuring Elastic Stack

Photo by Diana Parkhouse on Unsplash

So before we get into the grit of it all we need to start with an introduction to the Elastic Stack. But even before that, we need to start off with its predecessor the ELK stack

The ELK stack is an open-source project consisting of 3 Open Source Products, a winning combination of products that allows users to extract, parse, collect, view and Manage Logs in a convenient and efficient manner. The 3 technologies are …

1. Elasticsearch:- A search and storage Engine
2. Logstash:- Data extraction on parsing pipeline
3. Kibana:- Data viewing and Management Dashboard

So what happens is simple. You would use Logstash to point at the files, extract the data and parse it sends that data to an Elasticsearch cluster and then view it using Kibana Dashboard. Just don’t let the order confuse you ( ELK does allow me to put a better thumbnail than LEK I guess ? )

So how does this differ from Elastic Stack? Well, Logstash was a bit heavyweight being able to extract, parse and send Data in it’s streamed pipeline. So they swapped it to run parallelly with more a lightweight single-purpose Beats process (Filebeat for Log File tailing, MetricBeat for metrics data and so on) Additionally you could use different processes and stream data it to different (or the same) Logstash instances and extract data from multiple sources to the same instance. Which provides much more flexibility.

Source: https://communities.bmc.com/community/bmcdn/bmc_remedy_ar_system/blog/2018/07/31/using-the-elastic-stack-with-remedy-logs-part-1

Elastic Stack is quite popular because of the flexibility of its pipeline. With so many companies and industries going into the cloud with microservice architectures, a complex pipeline to extract data from your services is crucial. And that is where Elastic Stack comes in. Paired with cloud orchestration systems like Kubernetes, Elastic Stack is a very handy tool for extracting data.

Next Steps

So of course, we won’t be starting with the Kubernetes right away. But slowly dive into it by starting with the basics and diving deeper into its complexities.

So first let’s start with a Simple Example. Which I will dive into in this series of articles. But first, we need to configure the Elastic Stack first. So in the next tutorial, I will start with configuring the Elastic Stack. Starting with Logstash and Filebeat