ELK Stack Post — 1

Deploying ELK Stack for Apache Logs Analysis

In today’s world log analysis plays an important role. ELK Stack comes to rescue for this requirement. ELK is a popular log/data analysis open source platform. ELK stack has three main components:

From https://logz.io/learn/complete-guide-elk-stack/

Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack.
Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch.
Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack.

In this blogpost, our goal is to build ELK stack server in most possible simple way and analyze Apache Web Server logs.

Please note: When installing the Elastic Stack, you must use the same version across the entire stack. We are using 7.3.1 here. In this setup we are sending data directly to Elasticsearch without logstash, which is not a recommended approach. It can create inconsistency.

What do we need for this: A fair capacity server with Apache, Open-Jdk installed.

Elasticsearch Setup

# sudo mkdir /elk
# sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.1-linux-x86_64.tar.gz
# sudo tar -xzf elasticsearch-7.3.1-linux-x86_64.tar.gz
# cd /elk/elasticsearch-7.3.1/

Elasticsearch provides 3 files for configurational changes.

elasticsearch.yml for configuring Elasticsearch
jvm.options for configuring Elasticsearch JVM settings
log4j2.properties for configuring Elasticsearch logging

Update the ELK config file for this deployment.

# vi /elk/elasticsearch-7.3.1/config/elasticsearch.yml
network.host: 0.0.0.0
discovery.seed_hosts: ["127.0.0.1", "[::1]"]
cluster.initial_master_nodes: ["elkstack"]

Update the system settings to keep Elasticsearch happy.

# sudo vi /etc/sysctl.conf
vm.max_map_count=262144
fs.file-max = 10000
# sudo vi /etc/security/limits.conf
ubuntu    soft  nofile  6000
ubuntu    hard  nofile  65536

Start the services

# cd /elk/elasticsearch-7.3.1/bin
# sh elasticsearch
# sysctl -p

Access the service on

curl http://localhost:9200/

Kibana Setup

Download it and unzip:

# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.3.1-linux-x86_64.tar.gz
# tar -xvzf kibana-7.3.1-linux-x86_64.tar.gz

Configure the config file:

# vi /elk/kibana-7.3.1-linux-x86_64/config/kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Now Access the Kibana URL on:- http://<servername>:5601/

Kibana Dashboard

To proceed further we need Apache Setup as well. Please update apache config file to capture trace data in logs. Following setting will force Apache to log client IP addresses.

# In /etc/apache2/apache2.conf
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{ms}T %p %{Host}i" vhost_combined

Apart from this I have generated random load to generate access logs for analysis, using curl script and multiple browsers. We will see this data in Kibana.

FileBeat

Beats are lightweight agents that are installed on client hosts to collect different types of data for forwarding into the stack. We are using Filebeat in this scenario.

# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.1-linux-x86_64.tar.gz 
# tar xzvf filebeat-7.3.1-linux-x86_64.tar.gz 
# cd filebeat-7.3.1-linux-x86_64/

Update config file:

# validate file for Elasticsearch and Kibana
/elk/filebeat-7.3.1-linux-x86_64/filebeat.yml
# Filebeat comes packaged with pre-built modules that contain the configurations needed to collect, parse, enrich, and visualize data from various log file formats. We will enable apache module for use.
cd /elk
./filebeat modules enable apache
# documentation: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache.html

Apache module and fileset has variables that you can set to change the default behavior of the module, including the paths where the module looks for log files. You can set the path in configuration or from the command line. For example:

/elk/filebeat-7.3.1-linux-x86_64/modules.d/apache.yml

cd /elk/filebeat-7.3.1-linux-x86_64
./filebeat setup

# run this to upload data
./filebeat -e

Check that data in received -

Now Lets go back to Kibana dashboard and search for apache.

Voila ! You can see that it has visualize some important information by default.

Kibana can read Client IP data from logs and can create Geo-Heat Map instantly !

Kibana Dashboard 1

Kibana Dashboard 2

Kibana Dashboard 3

If you are done with the setup and practice, you can clear out data via curl.

# to delete all the indexes
curl -X DELETE 'http://localhost:9200/_all'
# to check the stats
curl http://localhost:9200/_stats/
# list all index:
curl -XGET http://localhost:9200/_cat/indices?v
#delete index:         
curl -XDELETE 'localhost:9200/index_name'
#delete all indices:   
curl -XDELETE 'localhost:9200/_all'
#delete document
curl -XDELETE 'localhost:9200/index_name/type_name/document_id'

URL for post 2.
Happy Learning 😃!