What is ELK Stack? How to deploy ELK using Docker, AWS EC2 and AWS Elastic Search

ELK stands for Elasticsearch, Logstash, and Kibana. Each of these three tools are open-source and can be used independently. But they are also work well together providing a solution to the common problem, ie. efficiently store, search and visualize large text files or logs. All these three tools are from the same company Elastic.

ELK — Elasticsearch, Logstash, and Kibana

Let us look at these components and understand their roles.

Elasticsearch

This is central component of the ELK stack. Elasticsearch offers multi-node (scalable) distributed search and analytics engine. It stores and indexes your data centrally and provides REST API access to it. You can think of it as a database for text files.

Logstash

This is the input tool for Elasticsearch. Logstash can receive logs or text files from different sources, transform it, and send it Elasticsearch.

Kibana

Kibana gives a UI to Elasticsearch, using which you can visualize and navigate the data stored in Elasticsearch.

Three ways to launch the ELK Stack

I am not going into the details of how to use these three tools or even how to launch them as there are so many articles on it. I will summarize the three ways you can start the ELK Stack.

First Way: Use Docker — the easiest way to try!

As explained here, there us a Docker image that has all these three tools backed in! Run the below Docker command to start a Docker container with these ELK Stack image.

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

These the port mapping.

  • 5601 — Kibana web interface
  • 9200 — Elasticsearch JSON interface
  • 5044 — Logstash Beats interface (lets you connect with the filebeat utility running on remote machine to stream logs to this ELK stack)

Still better, we can instead run three containers, one each for the three tools using docker-compose as explained here.

version: '2'services:elasticsearch:
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
ports:
- "5000:5000"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
kibana:
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./kibana/config/:/usr/share/kibana/config:ro
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
networks:elk:
driver: bridge

Second Way: Use AWS Elasticsearch

This is the second easiest way and this gives us a production grade ELK Stack with load balancer etc.

AWS Elasticsearch is a fully managed service that has Logstash Elasticsearch, and Kibana builtin.

Image from https://aws.amazon.com/elasticsearch-service/

You can visit AWS console and launch your AWS ELasticsearch service.

Follow console dialog screens to create the service.

Third Way: Manually Install ELasticSearch

Andrew Puch has a nice article that describes how to manually install the ELK Stack here. Pasting the steps in here as well to get an overview of the process involved.

Step1: Installing Elasticserach 1.7.2 in Centos as root user.

sudo su
yum update -y
cd /root
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm
yum install elasticsearch-1.7.2.noarch.rpm -y
rm -f elasticsearch-1.7.2.noarch.rpm
cd /usr/share/elasticsearch/
cd /etc/elasticsearch
# Configure
nano elasticsearch.yml
cluster.name: awstutorialseries
cloud.aws.access_key: ACCESS_KEY_HERE
cloud.aws.secret_key: SECRET_KEY_HERE
cloud.aws.region: us-east-1
discovery.type: ec2
discovery.ec2.tag.Name: "AWS Tutorial Series - Elasticsearch"
http.cors.enabled: true
http.cors.allow-origin: "*"
#Start
service elasticsearch start

Step2:Installing Logstash 1.5.4–1

cd /root
wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
yum install logstash-1.5.4-1.noarch.rpm -y
rm -f logstash-1.5.4-1.noarch.rpm
#Config
nano /etc/logstash/conf.d/logstash.conf

input { file { path => "/tmp/logstash.txt" } } output { elasticsearch { host => "ELASTICSEARCH_URL_HERE" protocol => "http" } }
#Startservice logstash start

Step3:Installing Kibana 4.1.2

yum update -y
cd /root
wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
tar xzf kibana-4.1.2-linux-x64.tar.gz
rm -f kibana-4.1.2-linux-x64.tar.gz
cd kibana-4.1.2-linux-x64
#Config
nano config/kibana.yml
elasticsearch_url: "ELASTICSEARCH_URL_HERE"
Commands
nohup ./bin/kibana &

Browse Kibana port 5601, http://KIBANA_IP:5601/

Using the ELK Stack

If we can to test upload a single record (document) we can do this.

curl -XPUT elasticsearch_domain_endpoint/movies/_doc/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'

We can bulk upload sample data provided by AWS here. Some command line examples that I have tried are given below.

# Single record or document
curl -XPUT https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/movies/movie/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'
{"_index":"movies","_type":"movie","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}# I downloaded sample data from AWS as a file, bulk_movies.json
# will looks like this
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "2" } }
{"director": "Frankenheimer, John", "genre": ["Drama", "Mystery", "Thriller"], "year": 1962, "actor": ["Lansbury, Angela", "Sinatra, Frank", "Leigh, Janet", "Harvey, Laurence", "Silva, Henry", "Frees, Paul", "Gregory, James", "Bissell, Whit", "McGiver, John", "Parrish, Leslie", "Edwards, James", "Flowers, Bess", "Dhiegh, Khigh", "Payne, Julie", "Kleeb, Helen", "Gray, Joe", "Nalder, Reggie", "Stevens, Bert", "Masters, Michael", "Lowell, Tom"], "title": "The Manchurian Candidate"}
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "3" } }
{"director": "Baird, Stuart", "genre": ["Action", "Crime", "Thriller"], "year": 1998, "actor": ["Downey Jr., Robert", "Jones, Tommy Lee", "Snipes, Wesley", "Pantoliano, Joe", "Jacob, Ir\u00e8ne", "Nelligan, Kate", "Roebuck, Daniel", "Malahide, Patrick", "Richardson, LaTanya", "Wood, Tom", "Kosik, Thomas", "Stellate, Nick", "Minkoff, Robert", "Brown, Spitfire", "Foster, Reese", "Spielbauer, Bruce", "Mukherji, Kevin", "Cray, Ed", "Fordham, David", "Jett, Charlie"], "title": "U.S. Marshals"}
{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "4" } }
{"director": "Ray, Nicholas", "genre": ["Drama", "Romance"], "year": 1955, "actor": ["Hopper, Dennis", "Wood, Natalie", "Dean, James", "Mineo, Sal", "Backus, Jim", "Platt, Edward", "Ray, Nicholas", "Hopper, William", "Allen, Corey", "Birch, Paul", "Hudson, Rochelle", "Doran, Ann", "Hicks, Chuck", "Leigh, Nelson", "Williams, Robert", "Wessel, Dick", "Bryar, Paul", "Sessions, Almira", "McMahon, David", "Peters Jr., House"], "title": "Rebel Without a Cause"}
# Uploading the above file using the bulk optioncurl -XPOST https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/_bulk --data-binary @bulk_movies.json -H 'Content-Type: application/json'# Searching within ELasticserach via the API
curl -XGET 'https://search-demo-x2dfu6md3nt6d7jyzyr6ixndmq.us-east-1.es.amazonaws.com/movies/_search?q=mars'
{"took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"movies","_type":"movie","_id":"1","_score":0.2876821,"_source":{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}}]}}

AWS Elasticserach gives us the Kibana endpoint as well, which we can directly browse. eg: https://YOUR AWS ELASTICSEARCH URL/_plugin/kibana/

What is AWS Cloudsearch?

Online asnwer is AWS Cloudsearch is a tool created by Amazon with similar features which it not open-source.

Hope this quick article on the ways we can deploy an ELK Stack was useful. Thank you for your time, please do follow for more such tiny reference articles that can come handy!

Sreeprakash Neelakantan

Written by

AWS Certified DevOps Engineer & Solutions Architect Professional — Docker | Kubernetes | DevOps — Trainer | Running | Swimming | Cycling

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade