Experiment: Introducing the FluentBit exporter for OpenTelemetry
Introduction
I recently got the chance of working with OpenTelemetry as part of the LFX mentorship program, where I spent my time working on creating a FluentForward exporter that would allow users to export traces using an already existing Fluentd/FluentBit agent (using the Fluentd Forward protocol). In this blog, I want to elaborate on the work done and provide an example demonstrating end-to-end telemetry flow — from exporting traces to FluentBit agent to receiving it in the collector and uploading it to the telemetry backend. I will start by exploring the need for the project and providing the prerequisite knowledge required to understand the blog, so feel free to skip ahead if you are already familiar with OpenTelemetry.
Motivation behind the project
OpenTelemetry provides a set of APIs, libraries, agents, and collector services to capture traces and metrics from applications. The telemetry collected by the various SDK’s can either be exported directly to different backends using exporters, or to the OpenTelemetry collector, which receives telemetry data using receivers, processes it using processors, and sends the data to other sources using exporters; using pipelines to handle data flow. There are three main telemetry types that these pipelines operate on — logs, metrics, and traces. At the time of writing of this blog, OpenTelemetry does not provide a logging API and various language implementations, rather focusing on enriching/processing logs received by agents like Fluentd. This implies that the responsibility of log collection and processing is offloaded to third party agents and SDKs.
It is common to deploy a Fluentd/FluentBit agent along with applications to collect, parse, and export the logs to the backend of your choice. Since there is a receiver available in the OpenTelemetry collector that can receive data using the Fluentd Forward (i.e.,fluentforwardreceiver), we can use these agents to export log data to the collector, where we can process it further in the log pipeline. The aim of this project is to export trace data using these FluentBit instances, with the data flow as shown below. This will eliminate the need to install an OpenTelemetry collector on Nodes and allow reuse of already installed agents.
Since FluentBit is used to export logs, the current implementation of the fluentforwardreceiver, the collector component responsible for receiving and converting log data to OpenTelemetry format, will be unable to recognize the trace data received and will instead try to parse it as a log message. However, since our goal is to export span data to the collector, I modified the receiver so that it can parse the traces received and convert them into the format required by OpenTelemetry. This change was added as a new component that is an extension to the fluentforwardreceiver which can be installed both in the logs and traces pipelines.
FluentBit Exporter
Setting up the environment
Now that we know the purpose and the scope of the project, we can look at an example application demonstrating how the exporter can be installed and the collector configured to export the traces to the backend of your choice. If you want to follow along, make sure that you have FluentBit installed on your machine, which you can get from the official website. Let’s get started!
The first thing that we need to do is start the FluentBit instance and the OpenTelemetry collector. Note that the FluentBit instance is configured to listen for trace data sent by the exporter on port 24224 using the Forward plugin, and it makes use of the tail input plugin to read the logs generated by our application. We also define a parser file that contains the information necessary to parse the log messages and provide the path to it in the config file. The parser file just defines a regex rule that parses log messages into time and message key-value pairs. The FluentBit agent then logs the received data to the console using the stdout output plugin and also sends the data to a configured URL using the Fluentd Forward protocol, which in our case is set to the collector, running on port 8006.
The config files are as shown:
For the collector, we have enabled two pipelines (logs and traces) — first adding the configurations of the various components and then defining which components belong in which pipeline. You can read more about pipelines in the collector architecture. Thus we can see that the logging pipeline will receive the logs from the FluentForward receiver, pass it to the batch processor, and log the output to the console, while the traces pipeline does the same thing, but instead of just logging it to the console, also sends the trace data to Jaeger.
With these config files, we can now start the FluentBit instance, the Collector, and Jaeger using the following commands:
FluentBit instance:
$ fluent-bit -c ./fluent.confCollector:
$ ./bin/otelcol_linux_amd64 --config ../otel-config.yamlJaeger using docker:
$ docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 -p 14250:14250 jaegertracing/all-in-one:latestTo demonstrate that the collector is able to parse both logs and traces, we create an app.log file, in which you can simply add the following example entry:
05-01-21 13:27:09 - log messageOr run the following python snippet:
import logging
logging.basicConfig(filename='app.log', filemode='w+', format='%(asctime)s - %(message)s', datefmt='%d-%m-%y %H:%M:%S', level=logging.INFO)logging.info('log message')
Example Application
Now that we have these services running, we can execute the demo application.
In the example application, the first thing that we do is initialize the trace provider and register the exporter, using the following function:
We create the exporter and register it with the BatchSpanProcessor, which sends the span data batches to the registered exporters for processing. We then create the TracerProvider, a stateful object that holds configuration information and provides access to tracers that allow us to create spans. As a final step, we register it globally using the SetTracerProvider method.
You can read more about the Tracing API in the spec defined over at the opentelemetry-specification repo, or have a look at the golang implementation.
Now, taking a look at the main method, we initialize the tracer provider, create a tracer, and start a span, as follows:
The function subOperation(ctx) creates a child span since it is a suboperation of the main method. In that function, we similarly create a span and add a message to it.
Now, we can execute the program, and as expected, the span data shows up in the FluentBit process. We can also see the log file entry that has been parsed by FluentBit.
Now, when we see the collector process, we can see that the collector has received a log message and 2 spans:
We can also navigate to the Jaeger UI at http://localhost:16686/ and find the traces under the service fluentexample and see the traces exported:
And as expected, the trace has the spans with the log messages and the attributes set:
Summary
During the mentorship program, I created a FluentBit exporter for OpenTelemetry, which can be found here. I also created an extension of the fluentforwardreceiver, which parses the received data as logs or spans based on the tag, thereby allowing users to use this receiver in both the traces and the logs pipelines. This component can be found here, and all the work completed by me as part of the LFX mentorship program can be found in this GitHub repo.
This is an experimental project and OpenTelemetry is looking forward to your feedback on this scenario and implementation.
Thanks to my mentor, Sergey Kanzhelev for his patience and guidance throughout the project and Tigran Najaryan for review.
Author
Aditya Prajapati, a CSE undergrad from K.J. Somaiya College Of Engineering who lives on the internet as @Syn3rman
