Setting up a Load-balanced Logstash behind and AWS ELB

Logstash running on EC2 instances behind an AWS ELB.

Step 1: Instance Launch Template and Autoscaling Group

To create an autoscaling group we can start off by creating one of two things:

  1. Launch Configuration
  2. Launch Templates (newer)
Launch Template creation page
Adding User data to Launch Template
cd /tmp/
sudo yum -y install java
rpm -vi logstash-6.5.4.rpm
access_key=`curl | python -c 'import sys, json; print json.load(sys.stdin)["AccessKeyId"]'`
secret_key=`curl | python -c 'import sys, json; print json.load(sys.stdin)["SecretAccessKey"]'`
session_token=`curl | python -c 'import sys, json; print json.load(sys.stdin)["Token"]'`
mkdir /root/.aws/
echo "[default]
region=us-west-2" > /root/.aws/credentials
aws secretsmanager get-secret-value --secret-id My_Logstash_Configuration | python -c 'import sys, json; print json.load(sys.stdin)["SecretString"]' > /etc/logstash/conf.d/log_config1.conf
/usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip
sudo mkdir -p /usr/share/logstash/geoip/
sudo curl | tar -xz -C /tmp/
sudo curl | tar -xz -C /tmp/
sudo cp /tmp/GeoLite2-City*/GeoLite2-City.mmdb /usr/share/logstash/geoip/
sudo cp /tmp/GeoLite2-ASN*/GeoLite2-ASN.mmdb /usr/share/logstash/geoip/
sudo chown logstash:logstash /usr/share/logstash/geoip/
echo 'echo " /var/lib/logstash
queue.type: persisted
path.queue: /tmp/logstash/queue
queue.max_bytes: 2gb
path.logs: /var/log/logstash" > /etc/logstash/logstash.yml' | sudo -s
sudo service logstash start<role_name>

Step 2: ELB Configuration

You would have to create a Network Load Balancer from the available load balancers if you want to listen for beats (filebeat, heartbeat etc.).

Creating a target group for the network load balancer

Step 3: The Filebeats Configuration

The logstash configuration remains the same as it would in a single node logstash deployment. But the thing that changes is the configuration of the beats that connect to logstash.

hosts: [""]
ttl: 120



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store