How to apply ELK Stacks to startups
Let’s monitor Spring boot & nginx server logs with ELK Stacks
What made me write this?
I’m a member of Coinplug’s server development team. We used to check the log by entering the server through the ssh terminal which has a view full of white letters on a black background. If you try to retrieve the logs, you’ll see:
$ cat LOGFILE.log | grep ERROR
And you had to enter commands one by one, which made developers and operators who checked the log files very frustrated.
Since it is inconvenient for developers to check log files, important logs are often missed or misinterpreted. In order to prevent this from happening, a log monitoring system seems to be needed.
After finding out that startups usually use ELK stacks to monitor logs, I decided to write this post and show you how to apply ELK Stacks to a log monitoring system.
What is the ELK Stack?
The ELK Stack is a technology stack of three open source products: Elasticsearch, Logstash, and Kibana.
- Elasticsearch (E): a search and analysis engine
- Logstash (L): a pipeline that collects, converts, and sends multiple log files to Elasticsearch
- Kibana (K): a tool that visualize the stored data in elastic search
Simply put, Logstash collects the logs generated from the server and sends them to Elasticsearch. This organizes the logs and sends them to Kibana and, finally, Kibana visualizes the cleaned logs with GUI and presents them to the users.
How to install ELK Stack
Docker-compose
To simplify the process, install Elasticsearch, Logstash, and Kibana at once using docker-compose. The required docker-compose file is the following:
You also need two setup files as follows:
kibana.yml
kibana.yml
is a configuration file required to connect Elasticsearch and Kibana. It contains the Elasticsearch address to connect to, as well as the ID and PW to connect to Elasticsearch.
logstash.conf
logstash.conf
is another configuration file that defines the input and output of Logstash.
Logstash can receive input from various sources. It can receive files or a regular keyboard input (stdin). In this case, let’s presume we simply receive tcp input and define tcp ports and JSON codecs in the input
area. Since Logstash must receive the log and forward it to Elasticsearch, I put the Elasticsearch address, the connection information and the Elasticsearch index name where the log will be stored in the output section.
Once you complete the setup, place docker-compose.yaml
, kibana.yml
, logstash.conf
files just like the image below:
|_ docker-compose.yaml
|_ kibana
|_ config
|_ kibana.yml
|_ logstash
|_ pipeline
|_ logstash.conf
After that, please enter the following command stated below to install and run the ELK stack.
$ docker-compose up
If you see the following screen after connecting to http://localhost:5601
and enter the Elasticsearch ID and password, you are on the right track.
Monitoring examples
Now that the ELK Stack installation is completed, let’s generate the server logs and send them to ELK Stack for monitoring.
Because our team normally uses nginx as a reverse proxy server and uses spring boot as a WAS server, I am going to monitor the nginx and spring boot server.
Spring boot server log monitoring examples
First, I created a simple server because we need a sample server to generate logs.
The server above is a simple http server that returns the string associated with “Hello world” and “value”, upon GET /api/v1/hello/{value} request. In order to leave an intermediate log and intentionally leave an error log, an incoming value greater than 100 is input and causes RuntimeException
.
When a request comes in, the log will remain like this:
Now that the log has occurred, let’s send it to the ELK Stack. Logback makes the process of sending spring boot logs simple, but first you need to add Logstash logback encoders.
If you are using maven as a build tool, add them like this:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.11</version>
</dependency>
or add them like this if you are using gradle:
compile group: ‘net.logstash.logback’, name: ‘logstash-logback-encoder’, version: ‘4.11’
Now, let’s add the following logback-spring.xml
file to src.main.resources
.
You can monitor the log by restarting the spring boot server and entering Kibana’s Discover
page.
You can also select the field you want to monitor in the Available fields
on the left. You only need to put the log level, log message, and logger name.
You can also look for the log you want in the search field. Let’s search the error log:
Example of spring boot server log monitoring: Custom field monitoring
In the previous example, only fields defined by Logstash logback encoder, such as log level
, logger name
, and log message
, could be monitored. Then, what should I do to monitor user-specified values? Like for instance, what should I do to keep an eye on the value
changes that come from spring boot server as a HTTP request?
The custom field feature provided by Logstash Logback encoder allows the user to specify the values to be monitored directly. For example, if you want to monitor the value
that comes in as a request to the sample server, you only need to add one line:
The code below the mark, saying “HERE!”, is the code that logs the custom field. Although it’s only the integer that is logged on the code above, you can log various types like Map
, String
, Object
and so on. If you want to know other ways to log custom fields, please refer to this link.
Now, let’s restart the sample server and generate some logs by sending HTTP requests. The value field will appear in Available fields
on the left side of Kibana’s discover page and you can monitor a value once you select it.
The advantage of monitoring the value field is that it is easy to search. For example, if you filter your search with a condition of value>50
, you can find the log that meets the criteria.
Example of nginx server log monitoring
To send Nginx server logs to the ELK stack:
You can find the details in the Kibana Home page
> Add data
> Logs
> Nginx logs
. (The location may vary slightly depending on the Kibana version. Use 7.10.1 in this article)
Basically, all you have to do is follow the Kibana’s guide. In this process, filebeat.yml is written like this:
output.elasticsearch:
hosts: [“localhost:9200”]
username: “elastic”
password: “MyPw123”
setup.kibana:
host: “http://localhost:5601”
Now put your code on modules.d/nginx.yml.
But please keep in mind that you need to put the actual path of nginx access log in access.var.paths
and put the actual path of nginx error log in error.var.paths
. Once you’ve done that, set the index pattern at the top left of Kibana discover page to filebeat
*.
Now you can monitor nginx log.
Conclusion
So far, I’ve demonstrated the process of applying a log monitoring function. In this case, I used only one Elasticsearch node, but it is desirable to cluster multiple Elasticsearch nodes to prevent the overload in case the amount of logs to monitor increases.
*Thank you for reading my article. If you have any opinions or issues related to this blog, please contact me through email written below.
Email: contact@coinplug.com
Reference