Fluent Bit Essentials: A Guide to Setup and Configuration

Published in

techbeatly

6 min readMay 26, 2024

Introduction to Fluent Bit

Fluent Bit is a specialized event capture and distribution tool that handles log events, metrics, and traces. It is a lightweight and efficient data collector and processor, making it ideal for environments where resource consumption is a critical consideration.

This article was originally published in techinik

Fluent Bit is crucial in modern logging and monitoring architectures, particularly within cloud-native environments. It acts as an intermediary between log sources and log storage or analysis systems. Fluent Bit collects logs from various sources, processes and filters them as needed, and then forwards them to destinations such as Elasticsearch, Kafka, or cloud storage services.

Key features of Fluent Bit include:

Lightweight and High Performance: Fluent Bit is designed to be resource-efficient, with a small memory footprint and low CPU usage, making it suitable for edge computing and IoT devices.
Flexible Configuration: It offers a flexible configuration system that allows users to define inputs, filters, and outputs to tailor log processing to specific needs.
Extensive Plugin Ecosystem: Fluent Bit supports a wide range of plugins for input, filtering, and output, enabling integration with various systems and formats.
Scalability: Fluent Bit can be deployed in large-scale environments, from single-node setups to distributed systems and Kubernetes clusters.

Setting Up Fluent Bit

To install Fluent Bit on a Linux machine, you need to run the following command. This command automatically configures the YUM repository and installs Fluent Bit:

[root@fluentbit ~]# curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3917  100  3917    0     0   7446      0 --:--:-- --:--:-- --:--:--  7446
================================
 Fluent Bit Installation Script
================================
This script requires superuser access to install packages.
You will be prompted for your password by sudo.
[fluent-bit]
name = Fluent Bit
# Legacy server style
baseurl = https://packages.fluentbit.io/centos/$releasever/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
Fluent Bit                                                                                                              440  B/s | 866  B     00:01
Rocky Linux 9 - BaseOS                                                                                                  4.1 kB/s | 4.1 kB     00:00
Rocky Linux 9 - AppStream                                                                                               4.7 kB/s | 4.5 kB     00:00
Rocky Linux 9 - Extras                                                                                                  3.6 kB/s | 2.9 kB     00:00
Package fluent-bit-3.0.5-1.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

Installation completed. Happy Logging!
[root@fluentbit ~]#

Once Fluent Bit is installed, you can verify the installation with the following command:

[root@fluentbit ~]# fluent-bit --version
Fluent Bit v3.0.5
Git commit:
[root@fluentbit ~]#

Configuring Fluent Bit

Fluent Bit’s configuration is composed of three main sections: Inputs, Filters, and Outputs. Each section serves a different purpose in the log processing pipeline, allowing you to collect, process, and forward logs efficiently.

Configuration Structure

Inputs: Define where Fluent Bit collects logs from. Common input plugins include tail for reading log files and systemd for reading systemd journal logs.
Filters: Allow you to modify, enrich, or exclude log records before they are sent to the output.
Outputs: Specify where the processed logs should be sent, such as Elasticsearch, Kafka, or cloud storage services.

Examples of Basic Input Plugins and Their Configurations

Tail Input Plugin

The tail input plugin reads log files and is commonly used to collect logs from applications or system logs. Below is an example configuration:

[INPUT]
    Name        tail
    Path        /var/log/syslog
    Tag         syslog
    Refresh_Interval 5
    Rotate_Wait 30
    DB          /var/log/flb_kv.db
    Mem_Buf_Limit 5MB

Name: Specifies the input plugin name (tail in this case).
Path: The path to the log file(s) to be read.
Tag: A tag to identify the logs collected by this input.
Refresh_Interval: How often (in seconds) Fluent Bit checks for new data.
Rotate_Wait: Time to wait (in seconds) to read new files after rotation.
DB: Path to a database file to track file offsets.
Mem_Buf_Limit: Maximum buffer size before data is flushed to output.

Systemd Input Plugin

The systemd input plugin reads logs from the systemd journal. This is useful for collecting system logs on systems that use systemd. Here’s an example configuration:

[INPUT]
    Name        systemd
    Tag         systemd
    Systemd_Filter  _SYSTEMD_UNIT=nginx.service
    Read_From_Tail  true

Name: Specifies the input plugin name (systemd in this case).
Tag: A tag to identify the logs collected by this input.
Systemd_Filter: Filters logs by systemd unit (e.g., nginx.service).
Read_From_Tail: Starts reading from the end of the journal.

Filtering Logs

Filters are a critical component of Fluent Bit’s log processing pipeline. They allow you to modify and exclude log records based on specific criteria before forwarding them to the desired output. Filters help ensure that only relevant and properly formatted logs reach their destination, which can improve performance, reduce storage costs, and facilitate more efficient log analysis.

Importance of Filters

Data Enrichment: Filters can add additional information to log records, such as metadata or tags, to provide more context.
Data Reduction: By excluding unnecessary logs, filters can help reduce the volume of data sent to storage or analysis systems, saving bandwidth and storage space.
Data Transformation: Filters can modify log records to ensure they meet the format requirements of the destination system, improving compatibility and readability.
Selective Processing: Filters allow you to process only specific types of log records, which can be crucial in complex environments where different logs require different handling.

Basic Examples of Using Filters

1. Parser Filter

The parser filter is used to parse and restructure log records. For instance, you can use it to parse JSON logs or to apply a custom parser to a log field.

Example configuration:

[FILTER]
    Name       parser
    Match      *
    Key_Name   log
    Parser     json

Name: Specifies the filter plugin name (parser).
Match: Defines which logs this filter should apply to (using a tag or wildcard).
Key_Name: The key of the log record to apply the parser to.
Parser: The name of the parser to use (e.g., json).

2. Grep Filter

The grep filter includes or excludes log records based on matching criteria. This is useful for filtering logs that contain specific keywords or patterns.

Example configuration to include only logs containing the keyword “ERROR”:

[FILTER]
    Name       grep
    Match      *
    Regex      log ERROR

Name: Specifies the filter plugin name (grep).
Match: Defines which logs this filter should apply to (using a tag or wildcard).
Regex: The regular expression to match log records that should be included.

Example configuration to exclude logs containing the keyword “DEBUG”:

[FILTER]
    Name       grep
    Match      *
    Exclude    log DEBUG

Exclude: The regular expression to match log records that should be excluded.

Sending Logs to a Destination

Fluent Bit uses output plugins to send processed log data to various destinations, such as log storage systems, databases, or analysis tools. Configuring these output plugins ensures that your logs are forwarded to the appropriate location where they can be stored, visualized, or further analyzed.

Configuring Output Plugins

Fluent Bit supports a wide range of output plugins for different destinations, including Elasticsearch, Amazon S3, Apache Kafka, and many more. Each output plugin requires specific configuration parameters to connect to and interact with the destination system.

Example Configuration for Sending Logs to Elasticsearch

Elasticsearch is a popular destination for storing and analyzing log data due to its powerful search and analytics capabilities. Below is an example configuration for sending logs to Elasticsearch:

[INPUT]
    Name        tail
    Path        /var/log/syslog
    Tag         syslog
    Refresh_Interval 5

[FILTER]
    Name        parser
    Match       syslog
    Key_Name    log
    Parser      json

[FILTER]
    Name        grep
    Match       syslog
    Regex       log ERROR

[FILTER]
    Name        record_modifier
    Match       syslog
    Record      hostname ${HOSTNAME}

[OUTPUT]
    Name        es
    Match       syslog
    Host        elasticsearch.local
    Port        9200
    Index       fluentbit
    Type        _doc
    HTTP_User   fluent_user
    HTTP_Passwd fluent_password
    tls         On
    tls.verify  Off

Name: Specifies the output plugin name (es for Elasticsearch).
Match: Defines which logs should be sent to this output (using a tag or wildcard).
Host: The hostname or IP address of the Elasticsearch server.
Port: The port on which Elasticsearch is listening (default is 9200).
Index: The Elasticsearch index where logs should be stored.
Type: The document type (deprecated in newer Elasticsearch versions but still required in some setups).
HTTP_User: Username for HTTP basic authentication (if required).
HTTP_Passwd: Password for HTTP basic authentication (if required).
tls: Enables TLS for secure communication.
tls.verify: Disables TLS verification (useful for testing but not recommended for production).

Setting up Fluent Bit involves configuring input plugins to collect logs, filters to process and enrich them, and output plugins to forward the logs to desired destinations. This modular approach allows you to build a highly efficient and customizable logging pipeline.

By following these guidelines, you can effectively use Fluent Bit to manage your logging needs, ensuring that your logs are collected, processed, and stored efficiently. Whether you are working with simple log files or complex distributed systems, Fluent Bit provides the flexibility and performance required for modern logging solutions. Stay tuned for more advanced configurations and use cases in our next blog post.

Feel free to reach out with any questions or share your experiences with Fluent Bit in the comments below. Happy logging!