ELK: Initiation

In this post, we are going to see the operation of ELK oriented to the log management in a Java application, and for this, we will use the application with Apache Camel that we have seen other times.

ELK is how it is known to a set of applications, specifically to: Elasticsearch, Logstash, and Kibana. All of them are open-source tools developed by Elastics. They allow you to collect data from any type of source and in any format to search, analyze and visualize the data in real-time.

• Logstash is a server-side data processing pipeline that consumes data from a multitude of different sources simultaneously. This data is transformed and sent to a data repository.
• Elasticsearch is a search and analytics engine for the data you have stored.
• Kibana allows us to create graphs and dashboards for users, based on the information Elasticsearch has.

Currently, this stack of applications is also composed by Beat. But we will not talk about it in this post.

Throughout the post, we will see how we can assemble it, how to configure it, and exploit the information of a Java application. The simplest method to have the stack available is through Docker Compose.

We start with Elasticsearch. It is the most important component and the one that needs more resources. In productive environments, it is recommended that a cluster is created to maintain a high availability and full performance. But for our example, as the main feature, it will have only one node. Otherwise, it will be a common Docker configuration:

• We will indicate a local folder as the container volume, which will allow us to keep the information even if we delete the container.
• A common network to be used by the containers.
• The exposed ports: 9200 and 9300.
• A health check to check if the instance is still working correctly. To do so, we will simply check that the exposed port gets a response when invoked.

The next section is the creation of the Kibana container. This tool will read the information stored in Elasticsearch, so it has to know where it is located. But this will not be indicated in the Docker Compose, which for this tool will have a very basic configuration. But it will be indicated in the YAML configuration file, which we indicate as input volume.

In the configuration file is where we must indicate the name of the service, its IP address, and, most importantly, where Elasticsearch is located.

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
## X-Pack security credentials
elasticsearch.username: elastic
elasticsearch.password: changeme

To finish the configuration of our Docker Compose, we only need to configure Logstash. This configuration will be a little more complex and consists of the following sections:

• A volume to indicate the file in which we will configure Logstash.
• A volume where we will configure the different pipelines that will serve as the input of the data to be sent to Elasticsearch by Logstash.
• A volume where Logstash will have access to the logs created by the application.
• The ports that the service exposes.
• The memory configuration of the JVM to be used.

The logstash.yml file will allow us to configure the tool. Which will not contain many values, for this example. As for Kibana, we will set our IP and where Elasticsearch is located:

http.host: 0.0.0.0
xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"]
## X-Pack security credentials
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme

The pipeline folder will store different configuration files that will allow Logstash to manage different sources of information and send them to Elasticsearch. These files are divided into three sections:

• Input: Where the source of the data is indicated. In our case, it will be the log files generated by the application. Those that were mounted on the volume.
• Filter: Where we can perform any filtering operation on the data. For our example, we will not perform any.
• Output: Where we indicate the repository to which we are going to send it. In this case Elasticsearch.

Once we have the Docker Compose configured, we can start it and check that it works correctly by accessing the path http://localhost:9200.

The next step will be the configuration of our application. The most important thing about this is that we will use logback. And that through the LogstashEncoder class we will be allowed to indicate certain characteristics. Such as the output format, add custom fields, or exclude certain packages.

We will start the application and perform several queries, for example to http://localhost:9090/book/1.

The next step will be to configure Kibana to be able to visualize all the data. To do this we access http://localhost:5601/app/home. Once inside we have to perform two steps.

• Create an index

We create an index, accessing first to the Manage Spaces menu option and then in the left menu to the Index Management option. Once in this screen, we create the index associated to the identifier that we indicated in the logstash pipeline.

• Viewing the information

From the Discover option of the main menu, we can access the logs that Logstash is sending to Elasticsearch.

With this, we will have already monitored our application through the logs. And we will be able to make better use of them through Kibana. We can search according to the application that sends us the information, the content of the message, or by date and time ranges. This is essential when your application has a certain size or produces a large number of messages.

If you want to see the whole example, you have it here.