ELK And Reverse Proxy Through Nginx Setup On RHEL Based System (ON SEPARATE MACHINE) → PART 2
The refined goal statement:
- Set up a monitoring system for an application running on four machines: Elasticsearch, Logstash, Kibana, and the application server.
- Install and configure Filebeat on the application server to send logs to Logstash.
- Deploy the react application on the application server using an Nginx reverse proxy.
- Configure Logstash to receive logs from Filebeat and forward them to Elasticsearch for indexing.
- Install and configure Kibana on the Kibana server, and set up an Nginx reverse proxy for Kibana.
- Establish the necessary connections between Logstash, Elasticsearch, and Kibana to enable log visualization.
- Create a graph in Kibana that displays the response codes (200, 502, 304) over time, using the indexed logs from Elasticsearch.
By achieving these goals, you will have a fully functioning monitoring setup that collects logs, indexes them, and provides visualizations in Kibana, including a graph showing the distribution of response codes over time.
Create four instances and configure security groups to define inbound and outbound traffic rules, allowing only authorized connections.
Securely SSH into the Application server to access the machine remotely.
Update the system
sudo yum updateInstall Filebeat:
Download the Filebeat RPM package:
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.0-x86_64.rpmInstall the RPM package:
sudo rpm -ivh filebeat-7.15.0-x86_64.rpmConfigure Filebeat: Edit the Filebeat configuration:
sudo nano /etc/filebeat/filebeat.yml#------Logstash Output------
output.logstash:
hosts: ["172.31.91.110:5044"]Start and enable Filebeat service:
sudo systemctl start filebeat
sudo systemctl enable filebeatYou can use “https://github.com/Piykmr/TextUtils.git” github repo to clone.
To get started with this repository, follow these steps:
1. Clone the repository to your local machine.
2. Once the cloning process is complete, navigate to the “TextUtils” folder.
3. Open a terminal or command prompt within the “TextUtils” folder.
4. Run the command
npm installin the terminal. This will install all the necessary dependencies for the project.5. After the installation is finished, run the command
npm start.6. The website will be launched and can be accessed by opening your preferred web browser and entering your IP address followed by port 3000.
Install Nginx:
Install Nginx:
sudo yum install nginxConfigure Nginx as a reverse proxy:
sudo nano /etc/nginx/nginx.confserver {
listen 80;
server_name _;
location / {
proxy_pass http://localhost:3000;
}
}we have to enable some module in filebeat so logs can be generated for specific module
sudo filebeat modules enable nginxNow, Use filebeat setup -e command to initialize and set up Filebeat with the predefined configurations. As, It creates the necessary index templates, dashboards, and other configurations in Elasticsearch and Kibana.
Here’s a breakdown of the command:
filebeat: This is the command used to execute Filebeat.setup: This subcommand initializes the setup process.-e: This flag stands for "enrollment" and is used to enable the Elasticsearch output and start the setup process immediately.
Securely SSH into the Logstash server to access the machine remotely.
Update the system
sudo yum updateIf Java is not installed, install it using the following command
sudo yum install java-1.8.0-openjdkInstall Logstash:
Download the Logstash RPM package:
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.15.0-x86_64.rpmInstall the RPM package:
sudo rpm -ivh logstash-7.15.0-x86_64.rpmConfigure Logstash: Create Logstash configuration file:
Below Logstash configuration is written to receive data from Beats (specifically the Beats input plugin) on port 5044 and send that data to Elasticsearch (using the Elasticsearch output plugin).
sudo nano /etc/logstash/conf.d/nginx.confinput {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => '%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][user_agent]}"' }
remove_field => "message"
}
}
output {
elasticsearch {
hosts => ["http://172.31.22.74:9200"]
index => "nginx-logs-%{+YYYY.MM.dd}"
}
}
Start and enable Logstash service:
sudo systemctl start logstash
sudo systemctl enable logstashSecurely SSH into the Elasticsearch server to access the machine remotely.
Update the system
sudo yum updateIf Java is not installed, install it using the following command
sudo yum install java-1.8.0-openjdkInstall Elasticsearch
Download the Elasticsearch RPM package
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-x86_64.rpmInstall the RPM package
sudo rpm -ivh elasticsearch-7.15.0-x86_64.rpmConfigure Elasticsearch: Edit the Elasticsearch configuration file
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#network
network.host: 172.31.22.74
http.port: 9200
#discovery
discovery.type: single-nodeStart and enable Elasticsearch service
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearchSecurely SSH into the Kibana server to access the machine remotely.
Install Kibana:
Download the Kibana RPM package:
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.0-x86_64.rpmInstall the RPM package:
sudo rpm -ivh kibana-7.15.0-x86_64.rpmConfigure Kibana: Edit the Kibana configuration file:
sudo nano /etc/kibana/kibana.ymlserver.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://172.31.22.74:9200"]Start and enable Kibana service:
sudo systemctl start kibana
sudo systemctl enable kibanaInstall Nginx:
sudo yum install nginxConfigure Nginx as a reverse proxy:
sudo nano /etc/nginx/nginx.confAdd the following configuration:
server {
listen 80;
server_name _;
location / {
proxy_pass http://localhost:5601;
}
}Test the Nginx configuration:
sudo nginx -t
Start and enable Nginx service:
sudo systemctl start nginx
sudo systemctl enable nginxFor rhel based systems: By default SeLinux is enabled (Enforcing) on all rhel based systems which will block all the traffic coming to it so in order to get the kibana accessible via nginx we can disable SeLinux.
Open the SELinux configuration file using a text editor such as vi:
sudo vi /etc/selinux/configLocate the SELINUX directive in the file. It will have one of the following values: enforcing, permissive, or disabled.
Change the value of SELINUX to permissive:
SELINUX=permissive
Save the changes and exit the text editor.
Again verify if value is set properly or not by below command
getenforceIf it is showing whatever you set then OK otherwise you can use below command
sudo setenforce 0That’s it! You have now installed and configured the ELK stack (Elasticsearch, Logstash, Kibana) on Red Hat Enterprise Linux (RHEL) along with Filebeat and Nginx as a reverse proxy for secure access. You can access Kibana by visiting http://your_domain_or_ip in a web browser.
/etc/logstash/conf.d/nginx.conf.
