Elasticsearch in the Cloud: How to Set Up on AWS EC2

Welcome back, cloud enthusiasts! In our journey through Amazon Web Services (AWS), we’ve explored the ins and outs of launching EC2 instances, mastered Spark for lightning-fast data processing, and harnessed NiFi’s data flow prowess. Today, we take an exciting turn into Elasticsearch — a powerful search and analytics engine that promises to revolutionize how we explore and understand our data.

Picture this: effortlessly finding the needle in the haystack, not just efficiently, but at lightning speed. That’s what Elasticsearch offers. With its distributed design, near-real-time indexing, and advanced search features, it’s a game-changer for developers and businesses alike. And what better platform to unleash its power than on AWS EC2 instances, running the latest Ubuntu 22.04?

In this guide, we’ll walk you through launching and setting up Elasticsearch on EC2 instances powered by Ubuntu 22.04. Whether you’re a seasoned developer looking to enhance your search capabilities or a data enthusiast eager to dive into distributed analytics, this guide is your ticket to success.

So, buckle up and get ready for an adventure as we unravel the mysteries of Elasticsearch on AWS EC2. By the end, you’ll have the knowledge and tools to build scalable, efficient search solutions that elevate your applications and data pipelines to new heights.

Join us on this journey as we demystify Elasticsearch — your gateway to unlocking the true potential of your data in the cloud.

Let’s start by ensuring our AWS EC2 security group settings are set up to allow traffic on the necessary ports for Elasticsearch. Begin by navigating to the AWS Management Console and selecting EC2. Once there, go to the Instances section and choose the relevant instance to view its details. Scroll down to find the Security Groups section and click on it to access the security group settings. In this section, you’ll need to adjust the inbound rules to accommodate Elasticsearch’s ports. Create a new rule for port 9200, a commonly used port for Elasticsearch, and specify a trusted IP address or range as the source, rather than allowing traffic from any IP address (0.0.0.0/0). This step ensures that incoming traffic on port 9200, necessary for accessing the Elasticsearch web interface, is limited to trusted sources. Once the inbound rules are updated, you’re ready to proceed with deploying and configuring Elasticsearch on your Ubuntu 22.04 EC2 instance.

After connecting to our EC2 instance, it’s essential to switch to the root user before launching Elasticsearch. This step grants superuser privileges, enabling us to perform administrative tasks required for installing and configuring Elasticsearch effectively. Once you’ve switched to the root user, you’re ready to proceed with the deployment process. To do this, execute the following command:

sudo su

After switching to the root user, the next step is to download Elasticsearch. It’s important to note that if you plan to use Kibana for visualization, ensure compatibility between Elasticsearch and Kibana versions before proceeding. You can check compatibility information at the following link: Elasticsearch-Kibana Compatibility Matrix. Failing to do so might lead to integration issues between Kibana and Elasticsearch.

By verifying compatibility in advance, you can avoid potential problems and ensure a smooth experience when integrating Kibana with Elasticsearch.

Now, let’s proceed with Elasticsearch 7.x. Use the following commands:

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --
dearmor -o /usr/share/keyrings/elastic.gpg

As the next step in launching Elasticsearch, add the Elasticsearch 7.x repository to your system by executing the following command. This command adds the Elasticsearch 7.x repository to your system’s list of package sources, enabling you to install Elasticsearch and related packages via package managers like apt.

echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Now, it’s essential to update your system to ensure that you have the latest package information. Execute the following command to update the package index. This command refreshes the local package index, allowing your system to fetch the latest package information from the configured repositories. It’s a necessary step before installing any new packages or updates, ensuring that you have access to the most recent versions available.

apt update

Next, install Elasticsearch on your system by executing the following command.

This command installs the Elasticsearch package, bringing the powerful search and analytics capabilities of Elasticsearch to your system. Once installed, Elasticsearch will be ready to use, but further configuration may be required depending on your specific use case. After the installation is complete, you can start and enable the Elasticsearch service to ensure it runs automatically on system boot. We’ll cover the configuration and startup process in the subsequent steps.

sudo apt install elasticsearch

The next step involves reloading the systemd manager configuration to ensure that any changes made to unit files are recognized. Execute the following command.

This command instructs systemd to reload its configuration files, making it aware of any recent changes. It’s often necessary after modifying unit files, such as service configuration files, to ensure that systemd manages the services correctly. After reloading the daemon, you can proceed to start and enable the Elasticsearch service, ensuring it runs smoothly on your system.

systemctl daemon-reload

The systemctl enable elasticsearch command is used to enable the Elasticsearch service to start automatically at boot time. By enabling the Elasticsearch service, you ensure that it starts automatically whenever the system boots up, providing uninterrupted access to Elasticsearch’s search and analytics capabilities. This ensures that Elasticsearch is always available, even after system reboots or power failures. Once enabled, you can start the Elasticsearch service using the systemctl start elasticsearch command to immediately start using it without needing to reboot the system.

systemctl enable elasticsearch

Once Elasticsearch is installed, it’s essential to configure its settings to optimize performance and security. Navigate to the Elasticsearch configuration file by executing the following command:

vim /etc/elasticsearch/elasticsearch.yml

One of the critical steps in setting up Elasticsearch on your AWS EC2 instance is configuring its settings to ensure optimal performance and connectivity. In this step, I’ll guide you through configuring the elasticsearch.yml file with essential parameters.

In this configuration:

• network.host: 0.0.0.0 allows Elasticsearch to accept connections from any IP address, enabling access from both local and external networks.
• http.port: 9200 sets the HTTP port to 9200, which is the default port for client communication with Elasticsearch.
• discovery.seed_hosts: ["0.0.0.0", "AWS_EC2_PUBLIC_IP"] specifies the seed hosts for cluster discovery. Replace "AWS_EC2_PUBLIC_IP" with the public IP address of your AWS EC2 instance. This allows Elasticsearch nodes to discover each other within the cluster.

# Configure network host to allow connections from any IP address
network.host: 0.0.0.0

# Set the HTTP port to 9200 for client communication
http.port: 9200 

# Specify the seed hosts for cluster discovery
discovery.seed_hosts: ["0.0.0.0", "AWS_EC2_PUBLIC_IP"]

Great job! You’re all set! Now, simply start Elasticsearch with the following command:

systemctl start elasticsearch

After that, you can access your Elasticsearch database by navigating to this link according to your AWS EC2 Public IP: https://AWS_EC2_PUBLIC_IP:9200

Congratulations on completing the setup! You’re now ready to unleash the power of Elasticsearch on your AWS EC2 instance.

As we wrap up this guide, you’ve now embarked on a journey to harness the immense power of Elasticsearch on your AWS EC2 instance. With Elasticsearch deployed, you’re on the path to unlocking actionable insights from your data like never before. But this is just the beginning!

Stay tuned for my upcoming blog posts, where we’ll dive deeper into the world of data engineering and ETL. Next up, we’ll explore launching Kibana for stunning visualizations that bring your data to life, followed by a guide on setting up Superset for interactive data exploration. With Elasticsearch paving the way, and Kibana and Superset on the horizon, you’re poised to revolutionize the way you analyze and leverage your data.

Get ready to embark on an exciting journey of discovery and empowerment!