What is ELK and Installing ELK Stack.

What is ELK Stack?

The ELK Stack, comprising Elasticsearch, Logstash, and Kibana, is recognized as the leading open-source log analysis platform globally. Rapidly gaining traction over proprietary alternatives, the ELK Stack is becoming the preferred choice for organizations seeking efficient log analysis and management solutions.

ELK Stack:

•Elasticsearch: A powerful search and analytics engine that stores and retrieves data quickly and efficiently.
•Logstash: A data processing pipeline that ingests, transforms, and sends data to Elasticsearch for indexing.
•Kibana: A visualization tool that provides interactive dashboards for analyzing and visualizing data stored in Elasticsearch.

When integrated, Elasticsearch, Logstash, and Kibana create a comprehensive data analysis solution. Logstash gathers data from various sources, applying filters and transformations before transmitting it to Elasticsearch for efficient storage and retrieval.

Elasticsearch serves as the storage solution, offering rapid, real-time search and analysis capabilities. Kibana delivers an intuitive interface for visualizing and analyzing the data housed in Elasticsearch.

Overall, the ELK Stack is a robust tool for real-time data collection, storage, and analysis, applicable to a range of use cases including log analysis, security monitoring, and business intelligence.

Why ELK Stack is so popular?

The ELK Stack has gained significant popularity for several key reasons:

1. Open-Source Flexibility: Being open-source, the ELK Stack allows organizations to customize and extend functionalities to meet specific needs without incurring licensing costs.
2. Scalability: Elasticsearch is designed to handle large volumes of data, making it suitable for growing datasets and enterprise-level applications.
3. Real-Time Analysis: The stack facilitates real-time data ingestion, search, and visualization, enabling organizations to respond promptly to emerging insights and threats.
4. User-Friendly Visualization: Kibana provides intuitive dashboards and visualization tools, making data analysis accessible to users with varying technical expertise.
5. Comprehensive Data Handling: With Logstash’s ability to collect and process data from multiple sources, the ELK Stack offers a holistic approach to data management.
6. Strong Community Support: A vibrant community and extensive documentation provide resources for troubleshooting, best practices, and enhancements, fostering continuous improvement.

Advantages of ELK Stack:

1. Cost-Effectiveness: As an open-source solution, the ELK Stack eliminates licensing fees, allowing organizations to leverage powerful data analytics without substantial financial investment.

2. Scalability: The architecture of Elasticsearch supports seamless scaling to accommodate increasing data volumes, making it ideal for both small and large enterprises.

3. Real-Time Data Processing: The stack enables real-time data ingestion and analysis, facilitating immediate insights and timely decision-making.

4. User-Friendly Visualization: Kibana provides an intuitive interface for creating interactive dashboards, making complex data analysis accessible to users with varying technical backgrounds.

5. Comprehensive Data Integration: Logstash’s ability to collect and transform data from multiple sources allows for efficient data management, ensuring a unified approach to analysis across diverse datasets.

Installation and Configuration of ELK Stack on Kali Purple.

Elastic Stack Installation

• Installing Dependencies:

During the installation of dependencies, please ensure to securely document the password for the elastic user.

sudo apt-get install curl
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/elastic-archive-keyring.gpg
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo bash -c "export HOSTNAME=kali-purple.kali.purple; apt-get install elasticsearch -y"

• Covert to single-node setup (or replace fqdn name in initial_master_nodes list with IP Address:

sudo sed -e '/cluster.initial_master_nodes/ s/^#*/#/' -i /etc/elasticsearch/elasticsearch.yml
echo "discovery.type: single-node" | sudo tee -a /etc/elasticsearch/elasticsearch.yml

• Installing Kibana:

sudo apt install kibana
sudo /usr/share/kibana/bin/kibana-encryption-keys generate -q

Adding the necessary keys to the `/etc/kibana/kibana.yml` configuration file.

echo "server.host: \"kali-purple.kali.purple\"" | sudo tee -a /etc/kibana/kibana.yml

Ensure that `kali-purple.kali.purple` is exclusively mapped to `192.168.1.100` in the `/etc/hosts` file to bind Kibana to that specific interface.

Execute the command `sudo systemctl enable elasticsearch kibana — now` to enable and start both Elasticsearch and Kibana services immediately.

• Enrolling Kibana:

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Open your web browser and navigate to `http://192.168.1.100:5601`.

Paste the token provided earlier.

sudo /usr/share/kibana/bin/kibana-verification-code

Enter the verification code into Kibana when prompted.

This will initiate the configuration process for Elasticsearch to get started.

Enter the username as `elastic` and use the password displayed during the installation process.

• Enable HTTPS for Kibana to enhance security:

/usr/share/elasticsearch/bin/elasticsearch-certutil ca

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns kali-purple.kali.purple,elastic.kali.purple,kali-purple --out kibana-server.p12

sudo openssl pkcs12 -in /usr/share/elasticsearch/kibana-server.p12 -out /etc/kibana/kibana-server.crt -clcerts -nokeys
sudo openssl pkcs12 -in /usr/share/elasticsearch/kibana-server.p12 -out /etc/kibana/kibana-server.key -nocerts -nodes

sudo chown root:kibana /etc/kibana/kibana-server.key
sudo chown root:kibana /etc/kibana/kibana-server.crt
sudo chmod 660 /etc/kibana/kibana-server.key
sudo chmod 660 /etc/kibana/kibana-server.crt

echo "server.ssl.enabled: true" | sudo tee -a /etc/kibana/kibana.yml
echo "server.ssl.certificate: /etc/kibana/kibana-server.crt" | sudo tee -a /etc/kibana/kibana.yml
echo "server.ssl.key: /etc/kibana/kibana-server.key" | sudo tee -a /etc/kibana/kibana.yml
echo "server.publicBaseUrl: \"https://kali-purple.kali.purple:5601\"" | sudo tee -a /etc/kibana/kibana.yml

Now, let’s test the ELK Stack using HTTPS.

We have successfully configured the ELK Stack to operate over HTTPS.

The setup is now complete. Please adjust the ELK Stack configuration according to your requirements. Thank you.