Best practices for Logstash

Marek Hornak
Aug 15, 2019 · 5 min read

Most of us, working with elastic stack has come to the point where we have to make optimizations in a matter of getting better indexing throughput or lower query response time.

Image for post
Image for post

The Elastic stack is quite easy to set up and get running. On the other hand, it is a pretty complex science to get the most of it and keep it in good condition. In this article, we will take a look at some basic rules and best practices to keep your stack healthy and robust.

As we are talking about Elastic stack, we will cover Logstash in this article and Beats and Elasticsearch in the upcoming sections. Although Logstash is not as complicated as Elasticsearch itself, it is crucial for some use cases and also needs the same level of focus for its proper configuration.

Logstash work modus is quite simple, it ingests data, process them, and then it outputs them somewhere. Each of this phase requires different tuning and has different requirements. Processing phase heavily relies on raw processor power, while outputs rely on the speed of the system to which Logstash outputs data.

Monitoring

If you are not that lucky, you can still get the information about running logstash instance by calling its API — which in default listens on 9600.
For example to get statistics about your pipelines, call:

curl -XGET http://localhost:9600/_node/stats/pipelines?pretty

and you will get all info in json format:

To better understand json output (and also Monitoring UI) you have to set id field for each input/filter/output in your Logstash pipeline definition:

Pipelines

Configure pipelines in YAML file, which is load at Logstash startup. Example configuration with three pipelines looks like this:.

The example uses pipeline config stored in files (instead of strings). Quite long and complicated parsing definitions is better to split into multiple files. That’s why we are using wildcards in the path.config properties. Logstash than concatenates every file matching given expression in alphabetical orders — to avoid problems with the filter in a different order than expected, we name conf files with numbering prefix: 101-parsing.conf , 102-another.conf

Persistent queue

  • In the case of high-load (which can’t be processed in real-time), you don’t have to store data in your application. Transmit them immediately Logstash. Store them in a queue, and process them continuously. Not storing your log files on persistent disk storage is a huge benefit when your application runs in a containerized environment.
  • In a scenario when your application is under high-load, Logstash will hit its processing limit and tell Filebeat to stop sending new data. Filebeat stops reading log file. Only-place where your logs are stored then is in running container. In the case of a container crash, you can lose a portion of logs.
  • By using persistent queue on your Logstash “ingest” pipeline, the logs are transferred almost immediately into Logstash, written into a queue, and you won’t lose anything.
  • The safety net in case of Logstash failure — not processed data are still available and will be processed after it is back up

Workers

To set the number of workers, we can use the property in logstash.yml:

pipeline.workers: 12

The default value is equal to a number of host’s CPU cores. But as suggested in docs, it is recommended to increase the number of workers when we see that CPU is not fully used.

Batch settings

The positive (side-)effect of batch size setting is, that some plugins take it into the account and can optimize their operation. For example, elasticsearch output plugin writes all events from batch using _bulk API which is an efficient way how to put large portions of data into elasticsearch.

Be careful with groks

When dealing with log files parsing, we always recommend to log into JSON format and have most data in a structured form. The ideal situation is if you don’t have to use groks at all and leave all parsing to json filter plugin.

To back this up with some real-world data: we had a java application with log4j logging into a text file and grok that made the parsing — timestamp, level, classname and lot more parsing. There were also log lines with network requests and responses more than 1000 characters long. After changing log format to JSON, and storing network related items into separate JSON fields, the Logstash throughput rose 7 times!

Switching from this…
… to this can save you lot of CPU load — formatted for better readability, in reality, it is on one line

Java settings

  1. set xmx and xms to same value. This way your application will avoid very expensive heap resize operation
  2. set xmx to value, that garbage collector will not run too often

Especially, point number 2 is very hard to achieve and require a lot of effort to find optimal value, you can find useful tips & tricks in the official documentation.

Conclusion

ableneo Technology

Application Development, Data Science, Experience Platforms

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store