How to Send Application Logs to Elasticsearch Effectively
Three different architectures for sending application logs to Elasticsearch
Introduction
This article explores three different architectures for sending application logs to Elasticsearch, each with their own set of advantages and disadvantages:
- Application -> Filebeat -> Logstash -> Elasticsearch
- Application -> Filebeat -> Elasticsearch
- Application (Java) + Logstash-logback-encoder -> Logstash -> Elasticsearch
While there may be other ways of sending logs, in my experience combining these approaches can fulfill any business requirements.
1- Application -> Filebeat -> Logstash -> Elasticsearch
This is a standard method for sending logs to Elasticsearch because it provides you with a lot of control. You can even change the path of sending logs to another Logstash or Elasticsearch server. here’s the sample filebeat.yml configuration file:
filebeat.inputs:
- type: log
paths:
- "myapp.log"
output.logstash:
hosts: ["192.168.1.1:4561"]and the logstash.conf file for receiving Filebeat logs:
input {
beats {
port => "4561"
host => "0.0.0.0"
}
}
filter { }
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "app-logs-%{+YYYY.MM.dd}"
}
}Pros:
- Control: You have a lot of control over the path that logs take.
- No code modification required: You don’t need to change the application code to send logs.
Cons:
- Performance issues: There are several steps involved in storing logs, which can impact performance.
- Disk load: The application writes logs to a file and Filebeat reads that file. This can cause a heavy disk load when the number of application instances increases.
2- Application -> Filebeat -> Elasticsearch
With this approach, you can eliminate Logstash from your log-sending pathway and send logs directly to Elasticsearch. This method works well if your logs are in JSON format and have been cleaned.
Below is a sample filebeat.yml configuration file that reads log files and sends them to Elasticsearch:
filebeat.inputs:
- type: log
paths:
- "myapp.log"
output.elasticsearch:
hosts: ["http://192.168.1.1:9200"] Pros:
- Improved performance: By eliminating Logstash, you reduce one step in the log-sending pathway.
- No code modification required: You don’t need to change the application code to send logs.
Cons:
- Limited control over log modification: Logstash offers many tools for parsing and cleaning logs before storing them in Elasticsearch. Without it, you have less control over log modifications.
3- Application (Java) + Logstash-logback-encoder -> Logstash -> Elasticsearch
If you have a Java application, you can use this approach to send logs directly to the Logstash without needing any interpreter. It is indeed possible to send logs without requiring Filebeat or other beats.
To be able to send logs to Logstash, you need to configure your Java application. First, add this dependency to your pom.xml file:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>7.3</version>
<scope>runtime</scope>
</dependency>then you need to add a new appender to your logback.xmlfile like this:
<property name="STACK_TRACE_COUNT" value="15"/>
<property name="CLASS_NAME_LENGTH" value="40"/>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>192.168.1.1:4560</destination>
<addDefaultStatusListener>false</addDefaultStatusListener>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<pattern>
<pattern>{"app_name": "myapp", "app_version":"1.0.0", "hostname": "${HOSTNAME}"}</pattern>
</pattern>
<mdc/>
<timestamp/>
<message/>
<threadName/>
<logLevel/>
<callerData/>
<stackTrace>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>${STACK_TRACE_COUNT}</maxDepthPerThrowable>
<shortenedClassNameLength>${CLASS_NAME_LENGTH}</shortenedClassNameLength>
<rootCauseFirst>true</rootCauseFirst>
</throwableConverter>
</stackTrace>
</providers>
</encoder>
</appender>
<root level="${ROOT_LEVEL}">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="LOGSTASH"/>
</root>I’m using the LogstashTcpSocketAppender class instead of the UDP one because I need to ensure that none of the logs are lost during transformation while sending them to Logstash. However, you can use the UDP if this level of reliability is not a concern for you.
On the other hand, Logstash should be configured to receive the logs, and store them in a location such as Elasticsearch. Here’s the sample Logstash configuration that I’m currently using:
input {
tcp {
port => 4560
codec => json_lines
}
}
filter { }
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "operationlog-%{app_name}-%{+YYYY-MM-dd}"
}
}Pros:
- Performance improvement: By sending logs directly to Logstash without storing them on disk, this approach has less overhead compared to previous methods.
Cons:
- Limitation: The
logstash-logback-encodercan only be used with Java applications. - Code modification requirement: To use this approach, you’ll need to modify the application code.
Conclusion
You can combine all of the above approaches in a single architecture, like this:
If you want to send logs from your Java application, you can use logstash-logback-encoder to send them directly to Logstash without needing to store them on disk.
For cleaning up data in the pathway of Python or NodeJS application logs, you can use Filebeat and Logstash together. However, if your data is already in JSON format, there’s no need to use Logstash. You can simply send logs to Filebeat, which will then store them in Elasticsearch.
Feedback
If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me an email at mallakimahdi@gmail.com. I would greatly appreciate your contributions to help make this article better. If you enjoyed this post, be sure to follow me to stay updated on my latest articles.
