Faster logs delivering from Fluentd

How to tune your fluentd configuration to send logs more frequently

Published in

Eugene Obrezkov

3 min readJun 13, 2018

Recently, I’ve faced a problem with slow logs collection and their delivery to our logs aggregator — Graylog. So, I started looking into fluentd documentation and digging up its sources to understand, how fluentd collects logs and how it delivers them.

It’s not an article in regular sense, it’s more of a small tip…

Collecting logs

I should start from how fluentd collects logs. For these purposes, it has input plugins.

Input plugins say for themselves, they have only one job to do — collect logs from different sources and transform them into fluentd records. But, they don’t output them immediately. Instead, they are being collected into buffers.

Let’s take a tail plugin, which can collect logs from files by tailing them. For that plugin, I’ve used the following configuration:

<source>
  @type tail
  tag kubernetes.containers.*
  path /var/log/containers/*.log
  refresh_interval 2
  read_from_head true
  pos_file /var/log/fluentd-containers.log.pos
  rotate_wait 5
  enable_watch_timer true
  enable_stat_watcher false
  open_on_every_update false
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

What does it do? Actually, it just runs tail -F, but a little smarter. It can remember the last read position, supports different parsers, etc… As a result of this configuration, you will get a fluentd record for every single line in your log files.

All of them will be stored in fluentd buffer (which can be memory, a file, or anything else, if you have a plugin for that).

Delivering logs

When logs are parsed and pushed into the buffer, fluentd has intervals by which fluentd will go into the buffer and starts pulling logs from buffer and output them somewhere else. For this, fluentd has output plugins.

Output plugin is responsible for receivening the fluentd record, parsing it in an appropriate format for specified ouput (in our case Graylog) and delivering it via transport (http, udp, tcp, whatever…).

I’ll show an example for Graylog. Here is our configuration for outputting logs to Graylog:

<match **>
  @type copy

  <store>
    @type gelf
    host "#{ENV['GELF_HOST']}"
    port "#{ENV['GELF_PORT']}"
    protocol "#{ENV['GELF_PROTOCOL']}"
  </store>
</match>

The configuration says to its plugin and fluentd, that all the records from fluentd should be delivered to our Graylog server (GELF protocol).

Where was the slowdown?

Turns out, that input/output plugins and their configurations have nothing with slow delivering (by default).

The problem was in buffer configuration itself. And, turns out, that the default configuration for buffer has specified interval by which it should be enqueued for delivering.

For our purpose, in elastic.io, we need to deliver logs from specific containers as fast as possible. Turns out, that you can modify fluentd behavior and how it should work with the buffer. The directive for controlling it called buffer.

So, let’s modify our configuration for output plugin to force immediate sending of logs from the buffer:

<match **>
  @type copy

  <store>
    @type gelf
    host "#{ENV['GELF_HOST']}"
    port "#{ENV['GELF_PORT']}"
    protocol "#{ENV['GELF_PROTOCOL']}"
    <buffer>
      flush_at_shutdown true
      flush_mode immediate
      flush_thread_count 8
      flush_thread_interval 1
      flush_thread_burst_interval 1
      retry_forever true
      retry_type exponential_backoff
    </buffer>
  </store>
</match>

As you see, you can change the behavior of the buffer in specific output plugins. I set flush_mode to immediate, so right after fluentd record is pushed into the buffer, it will be enqueued for delivering to our Graylog cluster.