Monitoring API logs via NGINX using Elastic — Part 3

Install & Configure Elastic Agent for Data Collection on the Target Server

This article is Part 3 of the series “Monitoring API logs via NGINX using Elastic”. Part 1 : How to Install ELK Stack version 7.x on Rocky Linux 8
Part 2 : Install & Configure Beats for Data Collection on Rocky Linux 8 Server

So previously we have explored the data collection method using beats. While powerful, imagine if your architecture contains dozens of servers, pods, or even cloud service. Now, you must install those beats one by one, tinkering with the server, manage the configuration, and that doesn’t include the monitoring. You can imagine a nightmare scenario is brewing here…

Luckily, Elastic hears people’s cry and came up with the solution: Elastic Agent. In short, here come one ring to rules them all.

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, and more. A single agent makes it easier and faster to deploy monitoring across your infrastructure. Each agent has a single policy you can update to add integrations for new data sources, security protections, and more.

1. Install Fleet Server

We now have elastic agents that get rid of the problem of installing so many independent shippers, but now we back to the square one with the question how do we manage all these elastic agents themselves? Here come Fleet as the answer.

Fleet provides a web-based UI in Kibana for centrally managing Elastic Agents and their policies.

Image capture by Author

To install it, you can follow these guides from Elastic:

Add a Fleet Server | Fleet and Elastic Agent Guide [7.17] | Elastic

2. Install Elastic Agent

Elastic has made amazing documentation that you can follow to install elastic agent. Here is the link to those documentation.

Command to download elastic-agent

Install Fleet-managed Elastic Agents | Fleet and Elastic Agent Guide [8.5] | Elastic

Other useful source for configuring integration

Install and Enroll Elastic Agents to Fleet Manager in Linux - kifarunix.com

FAQ

1. When I tried to update elastic-agent policy via Kibana Fleet, the change doesn’t apply to the elastic agent

You must open port 6788 in the target server’s firewall to allow Fleet to communicate back to the Elastic Agent

From Elastic FAQ

Frequently asked questions | Fleet and Elastic Agent Guide [8.5] | Elastic

The command to do so are:

sudo firewall-cmd --add-port=6788/tcp --permanent
sudo firewall-cmd --reload

2. I can’t register elastic-agent to the Fleet

You must configure fleet.yml first before installing the elastic-agent so the agent can communicate with the fleet. To do so, edit the fleet.yml file in your extracted installation file and paste following configuration

agent:
  id: 850de8f2-4a34-4535-b19f-2d9e8d4fe12d
  logging.level: info
  monitoring.http:
    enabled: false
    host: ""
    port: 6791
fleet:
  enabled: true
  access_api_key: [ACCESS_API_KEY]
  protocol: http
  host: 10.240.200.135:8220
  hosts:
  - http://10.240.200.135:8220
  ssl:
    verification_mode: none
    renegotiation: never
  timeout: 10m0s
  reporting:
    threshold: 10000
    check_frequency_sec: 30
  agent:
    id: ""

3. Elastic agent won’t sent metrics

To send NGINX Logs your account need to have following cluster permissions:

• manage_ingest_pipelines
• manage_pipeline
• manage