☁️ How to Install and Configure ELK Stack on Huawei Cloud

Introduction 🚀

Hi everyone 😎, I have already explained the ELK Stack in a previous article. In this article , I will guide you through the process of installing and configuring the ELK Stack on Huawei Cloud. The ELK Stack consists of three open-source tools: Elasticsearch, Logstash, and Kibana, which together provide a powerful solution for collecting, processing, analyzing, and visualizing log data.

Prerequisites

1. Huawei Cloud Account

Services Used

1. Virtual Private Cloud(VPC)
2. Cloud Search Service(CSS)
3. Elastic Cloud Server (ECS)

Let’s get started.

Service Details 🔨

By following the steps below, you will be able to set up your own ELK Stack environment on Huawei Cloud.

Step 1: Create VPC and Security Group

To begin, we need to create a Virtual Private Cloud (VPC) and configure a Security Group. Follow these steps:

1. Log in to Huawei Cloud and navigate to the VPC page.

2. Create a new VPC and configure the settings according to the provided image.

VPC — Configuration

3. Once the VPC is created, proceed to the Security Group page.

4. Create a new security group or use the default one.

5. Make sure that port 9300 is allowed for inbound and outbound traffic.

Security Groups

Step 2: Create CSS Cluster

Next, we will set up the Cloud Search Service (CSS) cluster, which will manage the ELK Stack. Follow these steps:

1. Go to the CSS service on Huawei Cloud.
2. Click on the “Create Cluster” button to create a CSS cluster.
3. Configure the cluster settings based on your requirements.

CSS Cluster Configuration

4. Wait for the cluster creation process to complete. This may take a few minutes.

5. Once the cluster is successfully installed, you can view it in the Clusters tab on the CSS page.

CSS Console

6. In the Operations tab, click on the “Access Kibana” button to access the Kibana interface.

7. On the Kibana page, click on the “Add Data” button to select the data type you want to work with.

Kibana Add Data

8. Choose the appropriate data type for your use case, such as “Logstash logs.”

Logstash logs

9. The opened page will provide instructions on how to use Logstash.

10. Go back to the CSS clusters page and click on the “More” button. Then select “Access Cerebro.”

11. In the Cerebro interface, click on “Known Clusters” and enter the node address. Click “Connect.”

12. In the upper section, click on “Nodes” to view the created nodes.

13. Click on the “More” button and select “Create Index” to name the index.

Cerebro — Create Index

Step 3: Create ECS

In this section, we will create a virtual machine using Elastic Cloud Server (ECS) to serve as our data source. Follow these steps:

1. Go to the ECS page on Huawei Cloud.
2. Create a new ECS instance and configure the settings as follows:

ECS — Configuration

4. Wait for the ECS instance to be created.

5. Log in to the ECS instance and install Logstash by executing the following commands:

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.10.2-linux-x86_64.tar.gz

tar -xvf logstash-7.10.2-linux-x86_64.tar.gz

cd logstash-7.10.2

5. Open the logstash-simple.conf file for editing:

vim logstash-simple.conf

6. Modify the file with the following configuration:

input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
}

filter {
}

output {
  elasticsearch {
    hosts => ["CSS Cluster IP Adress-1","CSS Cluster IP Adress-2"]
    index => "CREATE_INDEX_NAME"
    document_type => "mytype"
  }
}

7. Save the file and return to the root directory.

8. Install Nginx on the ECS instance by executing the following commands:

yum install -y nginx
find / -name nginx
cd /usr/share/nginx
systemctl start nginx
systemctl status nginx

9. Copy the Elastic IP (EIP) of the ECS instance and paste it into the browser to verify the successful installation of Nginx.

10. Install Java on the ECS instance by executing the following commands:

yum install java-1.8.0-openjdk.x86_64
vim .bash_profile

11. Add the following line to the .bash_profile file:

PATH=$PATH:$HOME/bin

JAVA?HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.372.b07-1.el7_9.x86_64
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

export JAVA_HOME JAVA_HOME CLASS_PATH PATH

12. Save the file and run the command:

java -version

13. Execute Logstash with the following command:

./bin/logstash -f logstash-simple.conf

Conclusion 🥇

By following the steps outlined in this tutorial, you have successfully created a Virtual Private Cloud (VPC) and Security Group, set up a Cloud Search Service (CSS) cluster, created an Elastic Cloud Server (ECS) instance, and installed and configured the ELK Stack components. You are now ready to start collecting, processing, and analyzing log data using the ELK Stack on Huawei Cloud.

👓You can reach me from my Linkedin account for all your questions and requests.

References🎈

1- Huawei Cloud VPC

Progressive Knowledge_Virtual Private Cloud_Huawei Cloud

2- Huawei Cloud CSS

Progressive Knowledge_Cloud Search Service_Huawei Cloud

3- Huawei Cloud ECS

Progressive Knowledge_Elastic Cloud Server_Huawei Cloud