What is the ELK Stack & Setup ELK Stack.
- What is the Log Analysis?
It is the process of understand server generated records(logs) that mean we can understand how to perform our system .Log analysis is the better way for the future enhancement in the product or system. A log Includes of a series of messages in time-sequence that describe activities going on within a system. It can be monitor performance errors,security errors in our system.
There are lot of log management tool such as Spulnk,Loggly,logentries,ELK stack. We are implementing this log analysis using ELK Stack.
2. What is the ELK Stack?
This is a three open source tools such as Elastic search, logstash and kibana. File beats also known as the prospectors that is multiple lightweight data collectors.It is a best log shipper but it can’t transform the logs into easy-to -analyze structured data. that’s the part of logstash (logstash can filtering unwanted bits of data/ transform data to another format).
Filebeat, by default, sends data to Elasticsearch. Filebeat can also be configured to send event data to Logstash.
Logstash is a server‑side data processing pipeline that take data from multiple sources simultaneously,filter & transforms it, and then sends it to Elasticsearch.
Elasticsearch is a search and analytic engine. All logs stores as JSON format in Elasticsearch. it has Nosql database. It can be easy to search because there are all logs as index file format.
Kibana provides us to graphical view data in Elasticsearch.
3. How to install ELK Stack?
prerequisites:
- Ubuntu Server with 20.04 LTS
- JDK
- 2 CPU and 4 GB RAM
- Open Ports 9200, 5601, 5044
Follow the procedure as I mentioned below,
Install and Configure ElasticSearch:
- wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch — no-check-certificate | sudo apt-key add -
- sudo apt-get install apt-transport-https
3.echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
4.sudo apt-get update
5.sudo apt-get install elasticsearch -y
6.sudo nano /etc/elasticsearch/elasticsearch.yml
- Go to Network section and uncomment network.host and replace your system IP or localhost : network.host: localhost
- Go to Network section and uncomment http.port: 9200
- Add this one in Discovery section also: discovery.type: single-node
7.By default, JVM heap size is set at 1GB. We recommend setting it to no more than half the size of your total memory. Open the following file for editing:
sudo nano /etc/elasticsearch/jvm.options
8.sudo systemctl start elasticsearch
9.sudo systemctl enable elasticsearch
10.sudo systemctl status elasticsearch
Output:
11.Enter below command to check elasticsearch running by port 9200
curl -X GET “localhost:9200”
Output:
Install and Configure Kibana:
1.sudo apt-get install kibana
2.sudo nano /etc/kibana/kibana.yml
3.Uncomment the below lines
4.add this lines=>
security.showInsecureClusterWarning: false
5.sudo systemctl start kibana
6.sudo systemctl enable kibana
7.sudo systemctl status kibana
Output:
Install and Configure Logstash:
1.sudo apt-get install logstash
2.sudo nano /etc/logstash/conf.d/02-beats-input.conf
3.Insert the below lines
4.sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf
5.Insert the below lines
6.sudo systemctl start logstash
7.sudo systemctl enable logstash
8.sudo systemctl status logstash
Output:
Install and Configure Filebeat:
1.sudo apt-get install filebeat
2.sudo nano /etc/filebeat/filebeat.yml
3.Comment the below lines
4.Uncomment the below lines
5.sudo systemctl start filebeat
6.sudo systemctl enable filebeat
7.sudo systemctl status filebeat
Output:
8.sudo filebeat modules enable system
9.check that ElasticSearch is receiving datalog from filebeat (yellow is represented by index files that have log data)
curl -XGET http://localhost:9200/_cat/indices?v
Output:
Access Kibana Web Interface:
you can access kibana from following URL,Enter you server IP address in the place of “your-server-ip”.
