Powerful logging with Docker, Filebeat and Elasticsearch

To perform an efficient log analysis, the ELK stack is still a good choice, even with Docker.

The ELK stack is composed of the following components:

• Logstash , a component to collect, transform and enrich log entries.
• Elasticsearch , a database and search engine that stores and indexes data (here: log entries) in a scalable way.
• Kibana , a UI, to easily access the data in Elasticsearch.

Logstash is a flexible and powerful tool, but it is considered resource intensive.

Nowadays, Logstash is often replaced by Filebeat, a completely redesigned data collector which collects and forwards data (and do simple transforms).

ELK Stack with and without Logstash.

Here we’ll see how to use an unique Filebeat to catch all our microservices logs, and apply a simple transformation if our application logs are JSON formatted, and send them to Elasticsearch.

To deploy our stack, we’ll use a pre installed Linux Ubuntu 18.04 LTS with Docker CE 17.12.0, Elasticsearch 6.2.4, and Kibana 6.2.4.

In Linux, the Docker containers log files are in this location :
/var/lib/docker/containers/<container-id>/<container-id>-json.log

Wonderful, because we can use that to collect the Docker logs via Filebeat to enrich important Docker metadata and send it to Elasticsearch. A single Filebeat container is being installed on every Docker host.

Each of our Java microservice (Container), just have to write logs to stdout via the console appender. For Spring Boot this is the default setting.

Docker JSON File Logging Driver with Filebeat as a docker container

This fulfills the single responsibility principle: the application doesn’t need to know any details about the logging architecture and doesn’t have to worry about organizing log files. Filebeat is solely responsible for sending log entries to Elasticsearch.

The Docker Bind Mount directories include the directories /var/lib/docker (the Docker Log Files) and /var/run/docker.sock (needed to access the Docker metadata).

Filebeat configuration

Let’s see now, how you have to configure Filebeat to extract the application logs from the Docker logs ?

Example extracted from a Docker log file (JSON), and showing the log field content (also in JSON):

{“@timestamp”:”2018–06–11T15:49:18.439+00:00",”@version”:”1",”message”:”Channel ‘example-service-1.sample-message-output’ has 1 subscriber(s).”,”logger_name”:”org.springframework.integration.channel.DirectChannel”,”thread_name”:”main”,”level”:”INFO”,”level_value”:20000}

We need to decode the JSON of the log field and to map each field (such as timestamp, version, message, logger_name, …) to an indexed Elasticsearch field.

This decoding and mapping represents the tranform done by the Filebeat processor “json_decode_fields”.

Here is an excerpt of needed filebeat.yml configuration file :

# filebeat.yml

filebeat.prospectors:
- type: log
  json.keys_under_root: true
# Json key name, which value contains a sub JSON document produced by our application Console Appender
  json.message_key: log
  enabled: true
  encoding: utf-8
  document_type: docker
  paths:
# Location of all our Docker log files (mapped volume in docker-compose.yml)
    - '/usr/share/filebeat/dockerlogs/*/*.log'
processors:
# decode the log field (sub JSON document) if JSONencoded, then maps it's fields to elasticsearch fields
- decode_json_fields:
    fields: ["log"]
    target: ""
# overwrite existing target elasticsearch fields while decoding json fields    
    overwrite_keys: true
- add_docker_metadata: ~
 
...

# Write Filebeat own logs only to file to avoid catching them with itself in docker log files
logging.to_files: true
logging.to_syslog: false

...

Full filebeat.yml file is available here .

Filebeat deployment

To deploy a Filebeat container on each docker node, we’ll use a custom Dockerfile :

FROM docker.elastic.co/beats/filebeat:6.2.4
# Copy our custom configuration file
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
# Create a directory to map volume with all docker log files
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/

Full Dockerfile file is available here.

The deployment is done with docker-compose, we need a docker-compose.yml file :

version: "3.5"
services:
  filebeat:
    hostname: filebeat
# ** Here to build the image, you need to specify your own docker hub account :
    image: bcoste/filebeat:latest
    build:
      context: ./filebeat
#Deploy is needed if you want to deploy in a Swarm
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
    volumes:
# needed to persist filebeat tracking data :
     - "filebeat_data:/usr/share/filebeat/data:rw"
# needed to access all docker logs (read only) :
     - "/var/lib/docker/containers:/usr/share/dockerlogs/data:ro"
# needed to access additional informations about containers
     - "/var/run/docker.sock:/var/run/docker.sock"
  
volumes:
# create a persistent volume for Filebeat
  filebeat_data:

Full docker-compose.yml file is available here.

From your linux shell you can now build and deploy your Filebeat (example deployment in a Swarm) :

docker-compose build
docker stack deploy --compose-file docker-compose.yml filebeat

Java microservice : Logback configuration

With Spring Boot, to configure LogBack logger, you need to add a file logback.xml to your Java application resources folder :

<configuration>
    <appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>
    <root level="INFO">
        <appender-ref ref="jsonConsoleAppender"/>
    </root>
</configuration>

Here we specified a standard ConsoleAppender, with a LogstashEncoder needed to encore logs in JSON (to be decoded later by Filebeat’s decode_json_fields).

For the LogstashEncoder, you need to add a dependency in your build.gradle file (or pom.xml for Maven) :

runtime('net.logstash.logback:logstash-logback-encoder:5.1')

Full java microservice source code is available here.

Filebeat working

Once Filebeat stack and Microservice stack are deployed in Docker, the log entries will now be sent to Elasticsearch, Docker metadata will be added and all functional JSON log fields will be decoded, giving this result (for one log line) :

{
 “_index”: “filebeat-6.2.4–2018.06.11”,
 “_type”: “doc”,
 “_id”: “nUWJ72MBn4ncS914jMQH”,
 “_version”: 1,
 “_score”: null,
 “_source”: {
 “@timestamp”: “2018–06–11T15:49:18.439Z”,
 “stream”: “stdout”,
 “level_value”: 20000,
 “logger_name”: “org.springframework.integration.channel.DirectChannel”,
 “docker”: {
 “container”: {
 “id”: “24bccb529c053674968354d1dc033f6b10ed5aa040e55df56d26e438ceecd209”,
 “labels”: {
 “com”: {
 “docker”: {
 “swarm”: {
 “task”: “”,
 “node”: {
 “id”: “exlduiap7i6ri2s3nk1kwhzy9”
 },
 “service”: {
 “id”: “v8rbm0q6on8945atlcdsrsctb”,
 “name”: “example_service”
 }
 },
 “stack”: {
 “namespace”: “my_namespace”
 }
 }
 }
 },
 “image”: “10.110.4.127:5000/my_namespace/example-service:latest@sha256:6b405f160e768d91b8c03a878f8f93e2c36f889d5e547044045cbe22ef79e666”,
 “name”: “example_service.1.yj3upi91ynugl338g0tn00xnq”
 }
 },
 “time”: “2018–06–11T15:49:18.439967308Z”,
 “@version”: “1”,
 “message”: “Channel ‘example-service-1.sample-message-output’ has 1 subscriber(s).”,
 “level”: “INFO”,
 “offset”: 54442,
 “log”: “{\”@timestamp\”:\”2018–06–11T15:49:18.439+00:00\”,\”@version\”:\”1\”,\”message\”:\”Channel ‘example-service-1.sample-message-output’ has 1 subscriber(s).\”,\”logger_name\”:\”org.springframework.integration.channel.DirectChannel\”,\”thread_name\”:\”main\”,\”level\”:\”INFO\”,\”level_value\”:20000}”,
 “prospector”: {
 “type”: “log”
 },
 “beat”: {
 “name”: “filebeat”,
 “hostname”: “filebeat”,
 “version”: “6.2.4”
 },
 “source”: “/usr/share/filebeat/dockerlogs/24bccb529c053674968354d1dc033f6b10ed5aa040e55df56d26e438ceecd209/24bccb529c053674968354d1dc033f6b10ed5aa040e55df56d26e438ceecd209-json.log”,
 “thread_name”: “main”
 },
 “fields”: {
 “@timestamp”: [
 “2018–06–11T15:49:18.439Z”
 ]
 }
}

In Elasticsearch, an index template is needed to correctly index the required fields, but Filebeat do it for you at startup.

In Kibana, you’ll be able to exploit the logs in it’s dashboards. From the Discover tab if you have no results, try to add search criterias :

Conclusion

This is a great solution because we avoid :

• a Filebeat for each of our container ( the log processing should not be part of the technical application)
• a duplicate log file generated by a File Appender (instead of standard Console Appender)

And we have very rich logs with additional informations about docker containers.