ELK for centralise Logging
ELK for centralise Logging :
The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana. They are all developed, managed ,and maintained by the company Elastic.
- E stands for ElasticSearch: used for storing logs
- L stands for LogStash : used for both shipping as well as processing and storing logs
- K stands for Kibana: is a visualization tool (a web interface) which is hosted through Nginx or Apache
Role in centralise logging :
ELK provides centralized logging that be useful when attempting to identify problems with servers or applications. It allows you to search all your logs in a single place. It also helps to find issues that occur in multiple servers by connecting their logs during a specific time frame.
prerequisite before starting ELK setup with docker for micro service applications :
Recommendation is learn below tech stack
- Docker
- Docker compose
- Basic docker commands
- Basic knowledge of logging files and format
Setup Details :
- Get ELK config details from following url : https://github.com/mahesh-wabale/ELK_Logginging/tree/master/ELK_6.2.2 .
- start required docker services with docker-compose : #docker-compose up
- Verify docker logs and docker processes and check all docker application services up and running : #docker ps
- Verify kibana url from browser : http://localhost:5601
- Verify Elastic search url from browser . http://localhost:9200/
For Enabling logging config lets perform following steps :
- Create Index pattern (* and timestamp filter).
2. Verify logs by clicking on Discover Tab .
cool your setup is now ready and you can push logs into ELK framework using filebeat . About publishing logs to ELK framework you need to learn about filebeat , grok patterns , logstash conf syntax and logging file format details .
I am planning to create another document about how to push logs into ELK setup and how to get loggs details on Kibana dashboard with filebeat https://www.elastic.co/beats/filebeat , Where we need to put settings or grok format (https://grokdebug.herokuapp.com/) details in logstash to push data into ELK framework in real time .
Below Diagram shows details about how we are able to push application logs with filebeat in ELK framework in real time .
Happy Learning ,
Thanks . . .
