Fluentd — Simplified

If your apps are running on distributed architectures, you are very likely to be using a centralized logging system to keep their logs. The most widely used data collector for those logs is fluentd. In this post we are going to explain how it works and show you how to tweak it to your needs. We are assuming that there is a basic understanding of docker and linux for this post.

Basic concepts

Have you ever run tail -f myapp.log | grep "what I want" > example.log in bash? Well this is what fluentd does pretty well, tailing logs or receiving data of some form, filtering it or transforming it and then sending it to another place.

That line above can be broken in multiple pieces:

Input

tail -f myapp.log

Here we are tailing a file forever, every time something is added to the file it will be shown in the screen. This is called input plugin in fluentd, tail is one of them, but there are many more.

Filtering

| grep "what I want"

Here we are taking the output from the tail -f and filtering out only lines that contain the string what I want. In fluentd-land this is called a filter plugin.

Output

> example.log