What is ELK Stack? How to deploy ELK using Docker, AWS EC2 and AWS Elastic Search

ELK stands for Elasticsearch, Logstash, and Kibana. Each of these three tools are open-source and can be used independently. But they are also work well together providing a solution to the common problem, ie. efficiently store, search and visualize large text files or logs. All these three tools are from the same company Elastic.

ELK — Elasticsearch, Logstash, and Kibana

Let us look at these components and understand their roles.

This is central component of the ELK stack. Elasticsearch offers multi-node (scalable) distributed search and analytics engine. It stores and indexes your data centrally and provides REST API access to it. You can think of it as a database for text files.

This is the input tool for Elasticsearch. Logstash can receive logs or text files from different sources, transform it, and send it Elasticsearch.

Kibana gives a UI to Elasticsearch, using which you can visualize and navigate the data stored in Elasticsearch.

Three ways to launch the ELK Stack

I am not going into the details of how to use these three tools or even how to launch them as there are so many articles on it. I will summarize the three ways you can start the ELK Stack.

First Way: Use Docker — the easiest way to try!

As explained here, there us a Docker image that has all these three tools backed in! Run the below Docker command to start a Docker container with these ELK Stack image.

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

These the port mapping.

  • 5601 — Kibana web interface
  • 9200 — Elasticsearch JSON interface
  • 5044 — Logstash Beats interface (lets you connect with the filebeat utility running on remote machine to stream logs to this ELK stack)

Still better, we can instead run three containers, one each for the three tools using docker-compose as explained here.

version: '2'services:elasticsearch:
context: elasticsearch/
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
- "9200:9200"
- "9300:9300"
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
- elk
context: logstash/
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
- "5000:5000"
- "9600:9600"
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
- elk
- elasticsearch
context: kibana/
- ./kibana/config/:/usr/share/kibana/config:ro
- "5601:5601"
- elk
- elasticsearch
driver: bridge

Second Way: Use AWS Elasticsearch

This is the second easiest way and this gives us a production grade ELK Stack with load balancer etc.

AWS Elasticsearch is a fully managed service that has Logstash Elasticsearch, and Kibana builtin.

Image from https://aws.amazon.com/elasticsearch-service/

You can visit AWS console and launch your AWS ELasticsearch service.

Follow console dialog screens to create the service.

Third Way: Manually Install ELasticSearch

Andrew Puch has a nice article that describes how to manually install the ELK Stack here. Pasting the steps in here as well to get an overview of the process involved.

Step1: Installing Elasticserach 1.7.2 in Centos as root user.

sudo su
yum update -y
cd /root
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm
yum install elasticsearch-1.7.2.noarch.rpm -y
rm -f elasticsearch-1.7.2.noarch.rpm
cd /usr/share/elasticsearch/
cd /etc/elasticsearch
# Configure
nano elasticsearch.yml
cluster.name: awstutorialseries
cloud.aws.access_key: ACCESS_KEY_HERE
cloud.aws.secret_key: SECRET_KEY_HERE
cloud.aws.region: us-east-1
discovery.type: ec2
discovery.ec2.tag.Name: "AWS Tutorial Series - Elasticsearch"
http.cors.enabled: true
http.cors.allow-origin: "*"
service elasticsearch start

Step2:Installing Logstash 1.5.4–1

cd /root
wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
yum install logstash-1.5.4-1.noarch.rpm -y
rm -f logstash-1.5.4-1.noarch.rpm
nano /etc/logstash/conf.d/logstash.conf

input { file { path => "/tmp/logstash.txt" } } output { elasticsearch { host => "ELASTICSEARCH_URL_HERE" protocol => "http" } }
#Startservice logstash start

Step3:Installing Kibana 4.1.2

yum update -y
cd /root
wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
tar xzf kibana-4.1.2-linux-x64.tar.gz
rm -f kibana-4.1.2-linux-x64.tar.gz
cd kibana-4.1.2-linux-x64
nano config/kibana.yml
elasticsearch_url: "ELASTICSEARCH_URL_HERE"
nohup ./bin/kibana &

Browse Kibana port 5601, http://KIBANA_IP:5601/

Using the ELK Stack

If we can to test upload a single record (document) we can do this.

curl -XPUT elasticsearch_domain_endpoint/movies/_doc/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'

We can bulk upload sample data provided by AWS here. Some command line examples that I have tried are given below.

# Single record or document
curl -XPUT https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/movies/movie/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'
{"_index":"movies","_type":"movie","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}# I downloaded sample data from AWS as a file, bulk_movies.json
# will looks like this
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "2" } }
{"director": "Frankenheimer, John", "genre": ["Drama", "Mystery", "Thriller"], "year": 1962, "actor": ["Lansbury, Angela", "Sinatra, Frank", "Leigh, Janet", "Harvey, Laurence", "Silva, Henry", "Frees, Paul", "Gregory, James", "Bissell, Whit", "McGiver, John", "Parrish, Leslie", "Edwards, James", "Flowers, Bess", "Dhiegh, Khigh", "Payne, Julie", "Kleeb, Helen", "Gray, Joe", "Nalder, Reggie", "Stevens, Bert", "Masters, Michael", "Lowell, Tom"], "title": "The Manchurian Candidate"}
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "3" } }
{"director": "Baird, Stuart", "genre": ["Action", "Crime", "Thriller"], "year": 1998, "actor": ["Downey Jr., Robert", "Jones, Tommy Lee", "Snipes, Wesley", "Pantoliano, Joe", "Jacob, Ir\u00e8ne", "Nelligan, Kate", "Roebuck, Daniel", "Malahide, Patrick", "Richardson, LaTanya", "Wood, Tom", "Kosik, Thomas", "Stellate, Nick", "Minkoff, Robert", "Brown, Spitfire", "Foster, Reese", "Spielbauer, Bruce", "Mukherji, Kevin", "Cray, Ed", "Fordham, David", "Jett, Charlie"], "title": "U.S. Marshals"}
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "4" } }
{"director": "Ray, Nicholas", "genre": ["Drama", "Romance"], "year": 1955, "actor": ["Hopper, Dennis", "Wood, Natalie", "Dean, James", "Mineo, Sal", "Backus, Jim", "Platt, Edward", "Ray, Nicholas", "Hopper, William", "Allen, Corey", "Birch, Paul", "Hudson, Rochelle", "Doran, Ann", "Hicks, Chuck", "Leigh, Nelson", "Williams, Robert", "Wessel, Dick", "Bryar, Paul", "Sessions, Almira", "McMahon, David", "Peters Jr., House"], "title": "Rebel Without a Cause"}
# Uploading the above file using the bulk optioncurl -XPOST https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/_bulk --data-binary @bulk_movies.json -H 'Content-Type: application/json'# Searching within ELasticserach via the API
curl -XGET 'https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/movies/_search?q=mars'
{"took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"movies","_type":"movie","_id":"1","_score":0.2876821,"_source":{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}}]}}

AWS Elasticserach gives us the Kibana endpoint as well, which we can directly browse. eg: https://YOUR AWS ELASTICSEARCH URL/_plugin/kibana/

What is AWS Cloudsearch?

Online asnwer is AWS Cloudsearch is a tool created by Amazon with similar features which it not open-source.

Hope this quick article on the ways we can deploy an ELK Stack was useful. Thank you for your time, please do follow for more such tiny reference articles that can come handy!

Connecting SMEs (Subject Matter Experts) to businesses who need Help with Microservices, Machine Learning, Cloud, IaC, and DevOps and Training