Avoiding ECS Agent (Docker Container) Logs in Fluent Bit

When using Fluent Bit to collect container logs from an AWS ECS cluster, you may want to exclude logs from the ECS Agent container to reduce noise and focus on your application logs.

Photo by Markus Spiske on Unsplash

Here’s how you can achieve this:

Step-by-Step Guide:

1. Enrich Your ECS Agent Container Metadata: Ensure that the ECS Agent container is identifiable through a specific label or attribute. If not, use the container name or another distinct characteristic.
2. Modify Fluent Bit Configuration: Use Fluent Bit’s filtering capabilities to exclude logs dynamically.

Example Configuration:

1. Create Lua Script for enriching container metadata: Save the following Lua script as /etc/fluent-bit/docker_metadata.lua:

DOCKER_VAR_DIR = '/var/lib/docker/containers/'
DOCKER_CONTAINER_CONFIG_FILE = '/config.v2.json'
DOCKER_CONTAINER_METADATA = {
  ['container_name'] = '\"Name\":\"/?(.-)\"',
  ['container_image_sha'] =  '\"Image\":\"/?(sha.-)\"'
}

cache = {}

-- Gets metadata from config.v2.json file for container
function get_container_metadata_from_disk(container_id)
  local docker_config_file = DOCKER_VAR_DIR .. container_id .. DOCKER_CONTAINER_CONFIG_FILE
  fl = io.open(docker_config_file, 'r')

  if fl == nil then
    return nil
  end

  -- Parse json file and create record for cache
  local data = {}
  for line in fl:lines() do
    for key, regex in pairs(DOCKER_CONTAINER_METADATA) do
      local match = line:match(regex)
      if match then
        data[key] = match
      end
    end
  end
  fl:close()

  if next(data) == nil then
    return nil
  else
    return data
  end
end

function encrich_with_docker_metadata(tag, timestamp, record)
  -- Get container id from tag
  container_id = tag:match'.*%.(.*)'
  if not container_id then
    return 0, 0, 0
  end

  -- Add container_id to record
  new_record = record
  new_record['container_id'] = container_id

  -- Check if we have fresh cache record for container
  local cached_data = cache[container_id]
  if cached_data == nil then
    cached_data = get_container_metadata_from_disk(container_id)
  end

  -- Metadata found in cache or got from disk, enrich record
  if cached_data then
    for key, regex in pairs(DOCKER_CONTAINER_METADATA) do
      new_record[key] = cached_data[key]
    end
  end

  return 1, timestamp, new_record
end

2. Create Lua Script for excluding ECS Agent by Container Name: Save the following Lua script as /etc/fluent-bit/exclude_ecs_agent.lua:

function filter(tag, timestamp, record)
    if record["container_name"] == "ecs-agent" then
        return 0, timestamp, record
    end
    return 1, timestamp, record
end

3. Create Docker Parser /etc/fluent-bit/parsers.conf

[PARSER]
    Name       docker
    Format     json
    Decode_Field_As    json     log
    Decode_Field_As    json     body

4. Fluent Bit Configuration: Update your Fluent Bit configuration file (e.g., /etc/fluent-bit/fluent-bit.conf) to use the Lua script:

[SERVICE]
    flush        1
    daemon       Off
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name              tail
    Path              /var/lib/docker/containers/*/*.log
    Tag               docker.*
    Docker_Mode       On
    Docker_Mode_Flush 5
    Docker_Mode_Parser container_firstline
    Parser            docker
    Mem_Buf_Limit     50MB
    Refresh_Interval  10
    Rotate_Wait       30
    Read_From_Head    Off
    DB                /etc/fluent-bit/logs.db

[FILTER]
    Name              lua
    Match             docker.*
    Script            docker-metadata.lua
    Call              encrich_with_docker_metadata

[FILTER]
    Name              lua
    Match             docker.*
    Script            exclude_ecs_agent.lua
    Call              filter

Important Part: Docker_Modeis enabled in Fluent Bit, it processes Docker log files and extracts useful metadata from the logs, including the container ID.

Configuration Explained:

• Docker_Mode On: Enables Docker mode in Fluent Bit, which tailors the log parsing specifically for Docker container logs.
• Docker_Mode_Flush: Sets the flush interval for Docker mode.
• Docker_Mode_Parser: Specifies a parser for the first line of Docker logs.
• Tag: Assigns a tag to the log, incorporating te container ID.
• Tag_Regex: Regex pattern to extract parts of the log file name, which can include the container ID.

Conclusion

As a result, configuring Fluent Bit to exclude ECS Agent logs involves several essential steps that streamline log management and enhance focus on application logs. By leveraging Fluent Bit’s Lua scripting capabilities and Docker mode configuration, you can dynamically filter out noise from ECS Agent containers while enriching logs with pertinent metadata. This setup not only improves log readability but also optimizes monitoring and troubleshooting efforts within ECS clusters. Implementing these configurations ensures that only relevant logs are processed, contributing to a more efficient and effective log management strategy overall.

Thanks for reading. If you have questions or comments regarding this article, please feel free to leave a comment below.

Would like to get in touch? Reach me out on LinkedIn:

https://www.linkedin.com/in/muhammedsaidkaya/