Using Logspout as a Fluentd Log Driver replacement
TL;DR
Logspout is a great tool for log routing where as Fluentd is a great tool for log processing. However, if you’re using docker’s built-in Fluentd Log Driver, you might have come across a few limitations in the docker-ce edition such as the log splitting issue and being disallowed from viewing your logs locally via docker logs. This article explains how to combine the strengths of Logspout and also route your logs directly into Fluentd using a custom built Logspout-Fluentd module thus surpassing these limitations and giving you much more control over your logs.
What’s Logspout?
From the official documentation:
Logspout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs where you want. It also has an extensible module system.
The way it attaches itself to containers is by subscribing to docker log events and then routing it to whatever destination you want, in our case to Fluentd. But it’s not all that simple since Logspout doesn’t come with a “built-in” router for Fluentd but because it has an extensible module system, we can write our own module to route logs to Fluentd. In the next section, I will show you how to do exactly that.
Extending Logspout with a Fluentd module
Logspout has instructions on how to extend itself using custom modules here. You can also read up on example custom built extensions here. I’ve taken a similar but yet up-to-date approach using Go 1.11 Modules and a Multi-stage build that just includes a custom Logspout binary.
Based on the instructions, first off, we need to write a simple go module that does the following:
- Register itself to Logspout to receive Docker container log streams
- Process the log streams and forward it to Fluentd
The code to register the module to Logspout is simple as shown below. We simply define and register a new Fluentd Adapter as the first thing via init() when the package is imported.
Our NewAdapter constructor function should basically construct itself to communicate with Fluentd and send logs to it using the Fluentd protocol specification. The code below uses the fluent-logger-golang/fluent library to do just that.
The above code configures the Fluentd adapter to communicate and ingest logs to Fluentd. All that’s left now is to capture the log stream and send it.
Once the adapter logic is ready, we can now include it in our custom Logspout build by following the instructions mentioned above. At the very least, we need to include the following:
- modules.go that will import our adapter into the Logspout binary.
- build.sh to download Logspout code so that it can be re-built with our custom module.
- Dockerfile that puts it all together and instructs docker how to build our custom Logspout.
Steps 1 and 2 are initiated on ONBUILD COPY triggers as soon as we base our image from gliderlabs/logspout:master so we don’t have to explicitly copy it into our Dockerfile.
Here’s the modules.go that imports our Fluentd adapter logic:
And here’s the build.sh file that simply downloads the Logspout source code so it can be custom built.
And here’s the Dockerfile that does a multi-stage build and uses go 1.11 modules:
To build, simply run the following command:
>> docker build -t mycustomlogspout .Using Logspout with our Custom Built Fluentd module
After we’ve built Logspout with our custom built Fluentd module, we simply run the docker image with the following command:
# Example to run custom built logspout locally:>> docker run --rm --name="logspout" \
-p 24223:24223 \
-v /var/run/docker.sock:/var/run/docker.sock \
-e TAG_PREFIX=docker \
-e TAG_SUFFIX_LABEL="com.mycompany.service" \
-e FLUENTD_ASYNC_CONNECT="true" \
-e PORT=24223 \
-e LOGSPOUT="ignore" \
mycustomlogspout
./logspout fluentd://<FLUENTD_IP>:<FLUENTD_PORT>
The environment variable TAG_PREFIX will add a fixed tag prefix to every log entry in the stream forwarded to Fluentd. Similarly the TAG_SUFFIX_LABEL environment variable will add a dynamic tag suffix by picking the value from the docker label of your container. Other FLUENTD* specific environment variables relate directly to the fluentd log driver configurations described here.
For more environment variables and configurations settings, you can read more on how to use them here.
We can test that logs are actually captured by Logspout and forwarded to Fluentd using a test echo container. The following demonstration uses Fluent-Bit which is a more light-weight version of Fluentd but follows the same protocol specification.
Other features of Logspout
One nice feature of logspout is streaming of all or specific container logs on a host via a REST API. You can simply curl the logspout endpoint and you get nice color coded logs. For this to work, you need to set the PORT=<PORT_NUMBER> environment variable and publish that port on the Logspout container as shown above. Following is the command to see this in action:
>> curl localhost:24223/logsadmiring_ishizaka|hello world
eloquent_lovelace|hello world
elated_neumann|hello world
Other Notes
- I was inspired to use Logspout due to a problem I faced where our application debug logs in JSON were split into smaller chunks when they exceeded 16K bytes. The Fluentd log driver wasn’t yet able to handle this sudden change and neither were any of it’s plugins. Hence it was impossible to construct the logs back to its original form. Using Logspout, this became transparent for our pipeline.
- It might not be trivial to view logs locally or on one of your virtual hosts. Usually you would have a good sophisticated log processing pipeline in place, such as an ELK/EFK stack. In most other cases, ssh-ing into machines to actually view the logs is also not a good idea. So this solution is not intended as a guideline to be included as part of your pipeline, but it doesn’t hurt to have it either. It is however helpful in environments where you need fast debugging and testing for your applications.
- Also note that Logspout will only gather logs from containers that are started without the -t option and that are configured with a logging driver that works with docker logs (journald and json-file).
Further reading
You can read more on this solution and get the code from my github repo @dsouzajude/logspout-fluentd.
Thank you for reading. Would love to hear your feedback and what you think by commenting below.
