Installing the ELK-stack on Ubuntu 16.04 and understanding its services

There have been quite a few blog posts on installing and configuring the ELK stack, which were very helpful when I was trying to understand how to setup the ELK stack on a production server for the first time. In this article I’m going to be focusing on some of the challenges I faced while installing ELK-stack and how I set up ELK-stack and some of it other services (beats, elastic-apm and x-pack).

Preparing the system
Before installing the ELK-stack on any server instance, always check the RAM available to the server. The ELK stack will take up at least 2GB of RAM, so make sure your instance was created with 2GB of RAM. 
Then, prepare the system by installing java 8. Do note, that the openjdk version must be 1.8.* for the ELK-stack to install properly. If the version is 1.9.* you will be facing problems! Install java using the commands

sudo add-apt-repository ppa:webupd8team/java 
sudo apt install oracle-java8-installer

To check the version, run the command

java -version
The result would be something like this.
java version “1.8.0_162”
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

If however you have installed some other version of java, that can be rectified by removing java from your system by using the command 
sudo apt-get autoremove openjdk-<version>-jre 
where version is java version “1.<version>.0_162” from above.

Installing ElasticSearch
Here, we’ll be installing ElasticSearch using the deb package repository system. In this article, I’ll only show the commands required to install elasticsearch. If you want to know the details, please visit
https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

 > wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
> sudo apt-get install apt-transport-https
> echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
> sudo apt-get update && sudo apt-get install elasticsearch
> sudo nano /etc/elasticsearch/elasticsearch.yml and configure elasticsearch
> sudo systemctl enable elasticsearch.service
> sudo systemctl start elasticsearch.service
To check if elasticsearch was installed correctly, just enter the following curl command.
curl -X GET “localhost:9200/”
This will give a response like this.
{
"name" : "Cp8oag6",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
"version" : {
"number" : "6.4.0",
"build_flavor" : "default",
"build_type" : "zip",
"build_hash" : "f27399d",
"build_date" : "2016-03-30T09:51:41.449Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "1.2.3",
"minimum_index_compatibility_version" : "1.2.3"
},
"tagline" : "You Know, for Search"
}

Also, remember to properly setup the proxy server you’re using(nginx or apache), so that the instance is accessible from other servers!

Installing Kibana
To install kibana, run the following commands. For more details, visit https://www.elastic.co/guide/en/kibana/current/deb.html

> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -  
> sudo apt-get install apt-transport-https
> echo "deb
https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
> sudo apt-get update && sudo apt-get install kibana
> sudo systemctl daemon-reload
> sudo systemctl enable kibana.service

Configuring Kibana
The path of the kibana conf file is /etc/kibana/kibana.yml (N.B. The conf file may be located in other places for other OS).
For a minimal setup, we need to set the following parameters.

server.port: 5601
server.host: 0.0.0.0 # so that kibana can be accessed from outside
elasticsearch.url: "http://localhost:9200"

This should be enough to get your kibana up and running!

Understanding the ELK-stack
Before installing Logstash, let us give an in general description of how the ELK-stack works. Firstly, ElasticSearch is similar to a noSQL database where data is stored in indices defined in Elastic. We can see the indices in Index Patterns sections of Kibana -> Management. Using logstash, we can collect, parse and store logs in Elastic, and see the logs using Kibana. Kibana also provides many visualization tools for graphs and charts. In this case, the kibana end-point should be made open and protected via XPack (authentication by username and password), or made accessible only from specific IP addresses. The logstash endpoint should either be kept open or secured via ssl using XPack or only be accessed from specific ips. Thus, the basic structure of a distributed system would be to send logs from your project to logstash. In my case, I used beats to send my project logs along with some system logs. To collect and store these logs, I defined a .conf file for logstash which accepted beats inputs and sent the input to Elastic after parsing. Now, let us setup Logstash :)

Installing Logstash

 > wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
> sudo apt-get install apt-transport-https
> echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
> sudo apt-get update && sudo apt-get install logstash
> sudo systemctl enable logstash
> sudo systemctl start logstash


Then go to the path /etc/logstash/conf.d and write a conf file which will define which inputs to parse, and where to write them.
Sample beats conf file.
input {
beats {
port => "XXXX"
}
}
filter {
grok {
match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[
@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[
@metadata][type]}"
user => "username" # if elasticsearch is secured via xpack
password => "password" # """
}
}

Et voila! You have successfully configured logstash to accept input from beats, which will write the log into elasticsearch server and can be visualized from kibana. Before this however, please restart logstash and kibana services!

sudo systemctl restart kibana
sudo systemctl restart logstash

Knowing Filebeats
Filebeat is a lightweight shipper for forwarding and centralizing log data. It was installed in our project server (which is separate from logstash), Filebeat monitors the paths that we specified, and forwarded them to Logstash for parsing and passing to Elastic for indexing. Installing and setting up filebeats is relatively simple, just follow the instructions from here!
https://www.elastic.co/guide/en/beats/filebeat/current/setup-repositories.html
Configure filebeats from the following links!

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration.html
https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-starting.html

Securing ELK-stack via XPack
The complete setup of XPack will be better described by the official documentation compared to whatever I might write. https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
Securing ElasticSearch and Kibana is pretty straightforward.

> sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
> sudo /usr/share/elasticsearch/bin/x-pack/setup-passwords auto|interactive
> sudo /usr/share/kibana/bin/kibana-plugin install x-pack
Configure /etc/kibana/kibana.yml to add elasticsearch.username and
elasticsearch.password that you have recently setup.