Running Fluentd as a Daemonset in Kubernetes

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. Fluentd tries to structure data as JSON as much as possible: this allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations.

Fluentd architecture. Image credit: Fluentd.org

At giffgaff, we’ve chosen Fluentd as our data collector. We run Fluentd as a daemonset in our Kubernetes cluster. This setup guarantees the logs of all pods running in any of our nodes are collected and shipped to our Elasticsearch cluster. Have a look at the following article where I talk about it

Log Aggregation With EFK

Fluentd is deployed using Helm. We build our Docker image using the official image as a base, and adding some plugins on top of it that allow us to enrich our logs and parsing them correctly.

This is how our Dockerfile looks like:

FROM fluent/fluentd-kubernetes-daemonset:v1.7-debian-elasticsearch7–2
USER root
RUN fluent-gem install fluent-plugin-multi-format-parser
RUN fluent-gem…