Fluentd: add log path to the record

Let’s say you use fluentd as your logging layer. In particular, you use td-agent with tail plugin configured to watch some logs. Those logs are in folders that are created by some process, and they are named tasks of some kind. So, the structure of files on the disk looks like:

/cluster/task1/logs/*.log
/cluster/task2/logs/*.log

Your td-agent source conf is:

<source>
    type tail
    tag tasks.logs
    path /cluster/tasks/*/logs/worker.log
    pos_file …
    …
</source>

The setup works, and as developers submit new tasks to the cluster, all the logs are getting caught and send to whatever destination you chose.

The good thing about such setup is that you don’t need to change td-agent.conf when a developer submits new task, and it can keep up with dynamically added tasks. The bad thing is, however, you loose the information from which task logs originated.

One solution to this situation may be adding the path of a log to the record.

To do that, you need to add path_key entry to the source config:

<source>
    type tail
    tag tasks.logs
    path /cluster/tasks/*/logs/worker.log
    path_key tailed_path
    pos_file …
    …
</source>

That will add the path of the log file to the record. If you want to further extract something from the path (for example, you don’t want a full path, you want task name only), that can be done with record transformer.

With record config like:

<filter tasks.*>
   type record_transformer
   enable_ruby
   <record>
      log_file "${record['tailed_path']}"
      hostname "${hostname}"
   </record>
</filter>

… you will rewrite the path to log_file property. You can then use Ruby to further transform the entry.