Design centralize logging architecture using Filebeat →ElasticSearch → Kibana

Well, I am NOT talking about these logs :)

I am talking about digital logs , generated by multiple environments across many servers.

As I am working on APIs and micro services , logs are very important part of our programs.

Our problem was we were getting millions of API calls every day.Normally servers and networks devices working fine but if response is slow, it was hard for my team to find out exact issue and where api response process is stuck. Also With modern cloud infrastructure logs growing every minute in production, it’s crucial to get an end to end view of what’s going on inside the servers and network devices.Of course we are storing logs of each API requests & processes.

But the problem was logs were on each servers in log files.It is very time consuming to go through logs files on each server and find errors. Also you can not monitor live stream of API calls , not able to do analysis.you can not search massive amounts of data quickly, pinpoint issues in real-time with custom views

Our objective was API logs generate a massive amount of data, and this data may be coming from multiple environments across many servers. To ensure that this data isn’t lost and can be used effectively, they should be consolidated and centralized to a single storage location

That’s why I started to thinking about centralized logging architecture.

Here is how my logging architecture was before centralized :

I have searched for few logging architecture. There are many and each of them have their own pros and cons. You need to decide what fits into your requirement. I have selected filebeat , elasticsearch and kibana. This is opensource stack & we were in primary stage so just want to check how it works in large scale. It works like charm :)

Elastic

EFK — ElasticSearch, Filebeat,Kibana is an open source project.Beyond log aggregation, it includes ElasticSearch for indexing and searching through data and Kibana for charting and visualizing data. Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them to either to Elasticsearch or Logstash for indexing.

Pros:

Open source tools come with a lot of control

Quick and easy setup for an open source solution

Cons:

ElasticSearch has its own REST API as well as JSON templates

Here is suggested architecture using ELK stack :

But i was trying to keep less component in between log source and storage that’s why i have removed buffer system and logstatsh. Filebeat can directly send logs to elastic search so in my case logstash is not necessary

Here are the steps how to setup filebeat to send logs to elastic search. It is very simple

prerequisites : Elasticsearch installed and Kibana url should be ready

Try walking through the full Getting Started guide for Filebeat. There are instructions for Windows. Basically the instructions are:

1. Extract the download file anywhere.
2. Open PowerShell with administrator rights.
3. Move the extracted directory into Program Files.

PS > mv filebeat-5.1.2-windows-x86_64 "C:\Program Files\Filebeat"

4. Install the filebeat service.

PS > cd "C:\Program Files\Filebeat"
PS C:\Program Files\Filebeat> powershell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1

5. Edit the filebeat.yml config file and test your config.

filebeat.yml path would be : C:\Program Files\filebeat\

Test your config
PS C:\Program Files\Filebeat> .\filebeat.exe -e -configtest

6. (Optional) Run Filebeat in the foreground to make sure everything is working correctly. Ctrl+C to exit.

PS C:\Program Files\Filebeat> .\filebeat.exe -c filebeat.yml -e -d "*"

7. Start the service.

PS > Start-Service filebeat

If you need to stop it, use Stop-Service filebeat. You might need to stop it and start it if you want to make changes to the config.

Now you can check log entries & live streams into Kibana.