Creating dashboards with real time interactive visualisation using ELK stack

I believe audience who have come this far, would have their understanding on business intelligence clarified to a moderate level. Also, I hope, they are aware of how much interactive visualisation play significance while providing historical, current and predictive views of business operations.

ELK, which stands for (E)lasticsearch, (L)ogstash, (K)ibana is an open source technology stack that was not necessarily designed for this type of real time analytics use case. Elasticsearch forms the platform for indexing data, in the form of JSON documents. Logstash serves as the data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favourite stash.(In this case, its naturally Elasticsearch). Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Put it all together, and you have a way to import all of your data sources into a single searchable place. In this article, I would demonstrate how to install and use ELK stack for creating and visualizing interactive dashboards, in Ubuntu 16.04 platform.

I prefer ELK stack for near real-time interactive visualisation because of its minimal latency in data processing. But it has its own demerits, such as limitations in variety of visualizations. For demonstration purpose, I refer to a csv data (SampleData.csv)available at my github account. Download the repository(https://github.com/ArunGARV/ELK-Stack) for implementation. It contains daily data of commuters between stations in US.

ELK Stack Installation

Elasticsearch and Logstash require Java, so we will install that now. We will install a recent version of Oracle Java 8 because that is what Elasticsearch recommends. It should, however, work fine with OpenJDK, if you decide to go that route. Add the Oracle Java PPA to apt:

$sudo add-apt-repository -y ppa:webupd8team/java

Update your apt package database:

$sudo apt-get update

Install the latest stable version of Oracle Java 8 with this command (and accept the license agreement that pops up):

$sudo apt-get -y install oracle-java8-installer

Once the JDK has been successfully installed, close the terminal and restart the machine. Now that Java 8 is installed, let’s install Elasticsearch. Open a new terminal. Starting off with Elasticsearch installation, you need to download the Elasticsearch tar file with the following command in the terminal:

$curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz

Go to the home folder where the downloaded tar file is present and enter the following command to extract it:

$tar -xvf elasticsearch-6.3.2.tar.gz

It will then create a bunch of files and folders in your current directory. We then go into the bin directory as follows:

$cd elasticsearch-6.3.2/bin/
$./elasticsearch

Open a web browser, and browse localhost:9200

Open a new terminal for Kibana. The Linux archive for Kibana v6.5.1 can be downloaded and installed as follows:

$curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
$tar -xzf kibana-6.3.2-linux-x86_64.tar.gz
$cd kibana-6.3.2-darwin-x86_64/bin
$./kibana

Open a web browser and enter the URL — localhost:5601

Open a new teminal to install logstash plugin. Enter following command to download the .tar file for logstash.

$curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz

Extract the downloaded .tar file using the following command:

$tar -xvzf logstash-6.3.2.tar.gz
$cd logstash-6.3.2/bin/

This will extract the files and folder into the current working directory( Home directory). Navigate to the bin directory within the logstash-6.3.2 directory. Now you need to prepare the logstash config file to provide pipelined indexing of data from ‘SampleData.csv’ into Elasticsearch. Copy the csv file and the logstash config file into the logstash bin folder. Edit the config file:

1. Provide the right input logstash location

Save the config file and type the following command in the terminal:

$./logstash -f SampleDataConf.conf

Type URL — localhost:9600 from web browser and observer something similar:

Verify the data ingestion into the index ‘samplebigdata’ in Kibana Dev Tools. Go to the Kibana browser page, click Dev Tools in the left pane. Type the following:

GET /_cat/indices?v

‘sampledata’ should be displayed in the index list. Make sure the name of the index as provided in the logstash config file should not contain capital alphabets.

Before creating the dashboard for the ingested data, we need to create the index pattern. Click ‘Management’ from left pane, and then click ‘index pattern’. Type the index name (sampledata) to generate index pattern. Choose ‘@timestamp’ as the Time Filter field name. Now navigate to ‘Discover’ tab in the left pane. Choose the time range ‘last 1 hour’. It should display something similar as below:

Lets start creating the dashboard. Go to the ‘Visualize’ tab. Click on ‘Create a visualization’. Choose pie chart from the list. Select ‘split slices’. Selects ‘Terms’ from the drop down for ‘Aggregation’. Select ‘to_station_name.keyword’ from the drop down for ‘Field’. Click the run/play button.

Lets create a histogram on daily commuters to all the stations. Select a new visualization. Select a histogram. Select the ‘sampledata’ index. Select x-axis aggregation as Date Histogram. Select ‘StartTime’ as Field and ‘Daily’ as interval. Click run/play button. Save the visualization.

Create new similar visualizations from the visualizations options. Once the required visualizations are created, create a new dashboard, and add relevant visualizations into the dashboard. Use ‘Dark theme’ for better visibility.

These visualizations are interactive and hence the entire dashboard changes when we put filter to a single graph. This would allow decision-makers to drill down through the layers of detail. It also encourage users to explore and even manipulate the data to uncover other factors. Also, any change made in the csv file, would be reflected accordingly in the dashboard.