Monitoring API logs via NGINX using Elastic — Part 2

Install & Configure Beats for Data Collection on Rocky Linux 8 Server

Photo by Firmbee.com on Unsplash

This article is Part 2 of the series “Monitoring API logs via NGINX using Elastic”. To understand this article please read the part 1 here

Previously in the part 1 we have explored the Elastic Architecture we want to build and how to install ELK Stack on Rocky Linux 8 server. Now, is the time to collect the data from the target server. Elastic has already made tools special for this purpose which is Beats.

ELK Stack Architecture using Beats

In this article I’ll guide you to install and configure Beats in the target server. We’ll use Filebeat and Metricbeat as the example here. To see other type of beats you can read the offical Elastic documentation.

1. Set up Elastic Yum Repository

ElasticSearch repository doesn’t come as a default in rocky linux. So we need to configure our package manager settings to enable it. In this case I use yum.

• Install GPG Key for ElasticSearch RPM Package

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

• Set Elastic.repo in /etc/yum.repos.d

sudo nano /etc/yum.repos.d/elastic.repo

• Paste in the repo detail

[elasticsearch-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• Update all cache

sudo yum update

2. Set Up Filebeat

a. Install filebeat in the server using the following command

sudo yum install -y filebeat

b. Using a text editor, create etc/filebeat/filebeat.yml and add the following content:

filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml

setup.kibana:
    host: "<IP_SERVER_KIBANA>:5601"

output.elasticsearch:
    hosts: ["<IP_SERVER_ELASTICSEARCH>:9200"]

setup.dashboards.enabled: true

Replace the bold-italic text with the appropriate value.

c. You can enable Filebeat modules using this command

filebeat modules enable elasticsearch
filebeat modules enable kibana
filebeat modules enable nginx

You can check the list of enable and disable modules using this command:

filebeat modules list

d. Enable and start the filebeat service:

sudo systemctl enable filebeat
sudo systemctl start filebeat

3. Set Up Metricbeat

a. Create the following Metricbeat configuration:
File: etc/metricbeat/metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

setup.kibana:
  host: "<IP_SERVER_KIBANA>:5601"

output.elasticsearch:
  hosts: ["<IP_SERVER_ELASTICSEARCH>:9200"]

setup.dashboards.enabled: true

b. You can enable MetricBeat modules using this command

metricbeat modules enable elasticsearch

metricbeat modules enable kibana

metricbeat modules enable nginx

You can check the list of enable and disable modules using this command :

metricbeat modules list

c. Start and enable the metricbeat service:

sudo systemctl enable metricbeat
sudo systemctl start metricbeat

4. Configuring NGINX on Target Server

a. Open the “/etc/nginx/nginx.conf” file and add the following “location” block in between the “include” and “location /” line:

include /etc/nginx/default.d/*.conf;

    location /server-status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            allow ::1;
            deny all;
        }

    location / {
        }

b. Restart NGINX with command

sudo systemctl restart nginx

c. Configure nginx module for metricbeat “/etc/metricbeat/modules.d/nginx.yml”. The settings will look like this:

metricbeat.modules:
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s

  # Nginx hosts
  hosts: ["http://127.0.0.1"]

  # Path to server status. Default server-status
  server_status_path: "server-status"

d. Restart Metricbeat with command:

sudo systemctl restart metricbeat

5. Testing visualization with dummy request

To test and visualize your log and metric data into Elasticsearch and Kibana, you can use this command to create some dummy request into NGINX.

while true ; do n=$(( RANDOM % 10 )) ; curl "localhost/?$n" ; sleep $n ; done

This process may be terminated by entering “Ctrl+C” at the terminal to exit the request loop at the conclusion of this tutorial.

You should be able to see your log & metrics in Kibana

Filebeat NGINX Dashboard in Kibana

Then congratz you’ve finally succeed setting up FileBeat & MetricBeat on your Target Server! Now you’ll be able to monitor those server via Kibana. In the next part we’ll explore how to simplify the Data Extraction setup with a new tool provided by Elastic. So, stay tuned & curious!

FAQ

If you encounter error in accessing Elasticsearch and Kibana in the remote server, make sure you’ve allowed connection in the firewall by running these commands :

sudo firewall-cmd --add-port=<port ElasticSearch or Kibana>/tcp --permanent
sudo firewall-cmd --reload

The default value is 9200 for elasticsearch & 5601 for Kibana