Getting started with Filebeat

How to analyze log files using Filebeat and Elasticsearch

Photo by Lewis Kang'ethe Ngugi on Unsplash

Inspecting and analyzing system log files are a part and parcel of every IT system administrator’s day. A centralized logging system makes life easier for IT admins and helps identify and fix faults more efficiently, ELK stack can help you store your logging data centrally and analyze your log files.

In the previous articles, I gave an overview of the elastic stack and installed an elastic search on Linux. In this article, I’ll focus on Filebeat.

Filebeat is a light weight log shipper which is installed as an agent on your servers and monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or log stash for indexing

Filebeat has two components

1. Harvester — A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output.
2. Input — An input is responsible for managing the harvesters and finding all sources to read from

How Filebeat works

1. It starts with one or more inputs that look in the locations you’ve specified for log data.
2. For each log that Filebeat locates, it starts a harvester.
3. Each harvester reads a single log for new content and sends the new log data to libbeat,
4. Libbeat aggregates the events and sends the aggregated data to the output that you’ve configured for File beat.

Lab Setup

In this article, I’ll set up a single-node elastic search cluster(refer to this article) and two apache webservers. I have used the GCP platform to build my test lab since it offers $300 USD free trial credit but you can do it on your servers or any other public cloud platform as well.

Filebeat Installation

Note: You’ll need an existing elastic search cluster to store log data and Kibana to visualize this data. We will also need metric beat installed on this server.

1. Please read this article to help set up a single-node elastic search instance and Kibana.
2. Please read this article to help set up the metric beat.

We’ll use the APT repository method to install Filebeat.

Installation and configuration of Filebeat on ELK Server

Minimum software requirements for Filebeat installation on ELK server should already be met as we already have elastic search and Metricbeat installed on this server.

1. Validate logging configuration

To start with out file beat setup, we need to first validate that logging is correctly setup for various components of our lab

1.1 Elasticsearch

path.logs: /var/log/elasticsearch

1.2 Kibana

1.2.1 configure Kibana logging

logging.dest: /var/log/kibana
logging.rotate.enabled: true
logging.rotate.keepFiles: 7

1.2.2 Create log file for Kibana

touch /var/log/kibana
chown kibana:kibana /var/log/kibana

1.2.3 configure logging for Metricbeat

sudo nano /etc/metricbeat/metricbeat.yml
## In the logging section, configure the following
logging.to_files: true
logging.files:
path: /var/log/metricbeat
name: metricbeat
keepfiles: 7
permissions: 0644

1.2. Restart service for changes to take effect

sudo systemctl restart elasticsearch
sudo systemctl restart kibana
sudo systemctl restart metricbeat

2. Install Filebeat

apt-get install filebeat

3. Configure Filebeat

sudo /etc/filebeat/filebeat.yml
### Live Reloadding
reload.enabled: true
reload.period: 10s
### Setup dashboards
setup.dashboards.enabled: true
### Kibana
host: “localhost:5601”
### elasticsearch output
hosts: [“localhost:9200”]
protocol: “http”
username: “elastic”
password: “goodwitch”
### logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
### setup monitoring through metricbeat
http.enabled: true
http.port: 5067

4. Enable beat-pack if not already enabled

sudo metricbeat modules enable beat-xpack

5. Configure beat-expack

sudo nano /etc/metricbeat/modules.d/beat-xpack.yml
hosts: [“http://localhost:5067"]
username: “beats_system”
password: “avatar”

6. Validate Metricbeat service is running

systemctl status metricbeat

7. Start Filebeat

systemctl start filebeat

8. Enable modules

sudo filebeat modules enable elasticsearch
sudo filebeat modules enable kibana
sudo filebeat modules enable system

With our ELK server setup with Filebeat, it is time to move on to our webservers.

Installation and configuration of Filebeat on Web Servers

1. Download and install the public signing key

wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2. Install apt-transport-https package

sudo apt-get install apt-transport-https -y

3. Save directory definitions

echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

4. System Update

sudo apt-get update

5. Validate logging is configured for Metricbeat

sudo nano /etc/metricbeat/metricbeat.yml
logging.to_files: true
logging.files:
path: /var/log/metricbeat
name: metricbeat
keepfiles: 7
permissions: 0644

6. Restart service for changes to take effect

sudo systemctl restart metricbeat

7. Install Filebeat

apt-get install filebeat

8. Configure Filebeat

sudo /etc/filebeat/filebeat.yml
### Live Reloadding
reload.enabled: true
reload.period: 10s
### Setup dashboards
setup.dashboards.enabled: true
### Kibana
host: “localhost:5601”
### elasticsearch output
hosts: [“localhost:9200”]
protocol: “http”
username: “elastic”
password: “goodwitch”
### logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
### setup monitoring through metricbeat
http.enabled: true
http.port: 5067

9. Enable beat-pack if not already enabled

sudo metricbeat modules enable beat-xpack

10. configure beat-expack

sudo nano /etc/metricbeat/modules.d/beat-xpack.yml
hosts: [“http://localhost:5067"]
username: “beats_system”
password: “avatar”

11. Validate Metricbeat service is running

systemctl status metricbeat

## 12. start Filebeat

systemctl start filebeat

13. Enable modules

sudo filebeat modules enable apache
sudo filebeat modules enable system

14. Stack monitoring

You can view if your beats are set up correctly under the stack monitoring, you should now be able to see Filebeat listed under your beats

If you click on a beat and select one of the servers, you should be able to see some more statistics

15. Dashboards

If your system is set up correctly , you should be able to see a dashboard just like this one

ou

So, here’s how you can configure Filebeat to store your logging data centrally.

If you want to do some hands-on practice on Elaticsearch, here is the link to my youtube playlist where I show you how to can set up a lab setup with 2 apache servers feeding data to a single-node Elasticsearch cluster deployed on a google cloud platform or GCP.