Logging with Fluent Bit and Fluentd in Kubernetes, pt.3
Fluent Bit is a fast and lightweight log processor, stream processor and forwarder. It’s gained popularity as the younger sibling of Fluentd due to its tiny memory footprint(~650KB compared to Fluentd’s ~40MB), and zero dependencies — making it ideal for cloud and edge computing use cases.
This post is part 3 in a series of posts about logging using Fluent Bit and the Fluentd forwarder in Kubernetes, and it describes the steps to deploy Fluentd and Fluent Bit.
Other posts in this series:
- Part 1 Motivation and Architecture Overview.
- Part 2 Deploying a single-node Elasticsearch, along with Kibana and Cerebro.
- Part 4 Viewing the End Result.
Deploying Fluentd
Fluentd is the log aggregator and processor stage before Elasticsearch, and we will deploy this now. We will use the Bitnami Fluentd Helm chart.
- Extend the Bitnami image by installing the
rewrite_tag_filter
plugin. We will push this up to docker hub as a custom image, to be used later.
CUSTOM_DOCKER_IMG=georgegoh/fluentd:1.10.4-debian-10-r2-rewrite_tag_filter
cat <<EOF | docker build -t ${CUSTOM_DOCKER_IMG} -
FROM bitnami/fluentd:1.10.4-debian-10-r2
LABEL maintainer "Bitnami <containers@bitnami.com>"
## Install custom Fluentd plugins
RUN fluent-gem install 'fluent-plugin-rewrite-tag-filter'
EOF
docker push ${CUSTOM_DOCKER_IMG}
2. Add the Bitnami Helm repo.
helm repo add bitnami https://charts.bitnami.com/bitnami
3. Create a custom ConfigMap
that can send output to Elasticsearch. Substitute ES_HOST=es.lab.example.com
with your own Elasticsearch hostname.
ES_HOST=es.lab.example.com
cat <<EOF | sed "s/<elasticsearch-host>/${ES_HOST}/" > fluentd-elasticsearch-output-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-elasticsearch-output
namespace: k8s-system-logging
data:
fluentd.conf: |
# Prometheus Exporter Plugin
# input plugin that exports metrics
<source>
@type prometheus
port 24231
</source>
# input plugin that collects metrics from MonitorAgent
<source>
@type prometheus_monitor
<labels>
host \${hostname}
</labels>
</source>
# input plugin that collects metrics for output plugin
<source>
@type prometheus_output_monitor
<labels>
host \${hostname}
</labels>
</source>
# Ignore fluentd own events
<match fluent.**>
@type null
</match>
# TCP input to receive logs from the forwarders
<source>
@type forward
bind 0.0.0.0
port 24224
</source>
# HTTP input for the liveness and readiness probes
<source>
@type http
bind 0.0.0.0
port 9880
</source>
# Throw the healthcheck to the standard output instead of forwarding it
<match fluentd.healthcheck>
@type stdout
</match>
# rewrite tags based on which namespace the logs come from.
<match kube.**>
@type rewrite_tag_filter
<rule>
key \$['kubernetes']['namespace_name']
pattern /^(kube-system|kubeapps|k8s-system-[\S]+)$/
tag ops.\${tag}
</rule>
<rule>
key \$['kubernetes']['namespace_name']
pattern /.+/
tag prod.\${tag}
</rule>
</match>
<match ops.kube.**>
@type copy
@id output_copy_ops
<store>
@type elasticsearch_dynamic
@id output_elasticsearch_ops
host <elasticsearch-host>
port 9200
logstash_format true
logstash_prefix kube-ops.\${record['kubernetes']['namespace_name']}
<buffer>
@type file
path /opt/bitnami/fluentd/logs/buffers/ops-logs.buffer
flush_thread_count 2
flush_interval 5s
</buffer>
</store>
</match>
<match prod.kube.**>
@type copy
@id output_copy
<store>
@type elasticsearch_dynamic
@id output_elasticsearch
host <elasticsearch-host>
port 9200
logstash_format true
logstash_prefix kube.\${record['kubernetes']['namespace_name']}
<buffer>
@type file
path /opt/bitnami/fluentd/logs/buffers/logs.buffer
flush_thread_count 2
flush_interval 5s
</buffer>
</store>
</match>
EOF
kubectl apply -f fluentd-elasticsearch-output-configmap.yaml
4. Install the Fluentd Helm chart. Substitute the image.repository
and image.tag
values with the relevant values from step 1.
helm install fluentd \
--set image.repository=georgegoh/fluentd \
--set image.tag=1.10.4-debian-10-r2-rewrite_tag_filter \
--set forwarder.enabled=false \
--set aggregator.enabled=true \
--set aggregator.replicaCount=1 \
--set aggregator.port=24224 \
--set aggregator.configMap=fluentd-elasticsearch-output \
bitnami/fluentd -n k8s-system-logging
Consider using a Horizontal Pod Autoscaler for the fluentd
StatefulSet to react to higher volumes of incoming logs.
Now that we’ve completed setup of Fluentd, Elasticsearch and Kibana, it’s time to move on to Fluent Bit and complete the logging setup.
Deploy Fluent Bit
- Save the following in a file called
values.yaml
.
on_minikube: false
image:
fluent_bit:
repository: fluent/fluent-bit
tag: v1.3.7
pullPolicy: Always
# When enabled, exposes json and prometheus metrics on {{ .Release.Name }}-metrics service
metrics:
enabled: true
service:
labels:
k8s-app: fluent-bit
annotations:
'prometheus.io/path': "/api/v1/metrics/prometheus"
'prometheus.io/port': "2020"
'prometheus.io/scrape': "true"
port: 2020
type: ClusterIP
serviceMonitor:
enabled: false
additionalLabels: {}
# namespace: monitoring
# interval: 30s
# scrapeTimeout: 10s
backend:
type: forward
forward:
host: fluentd-0.lab.spodon.com
port: 24224
tls: "off"
tls_verify: "on"
tls_debug: 1
shared_key: thisisunsafe
parsers:
enabled: true
## List the respective parsers in key: value format per entry
## Regex required fields are name and regex. JSON and Logfmt required field
## is name.
regex:
- name: cri-mod
regex: "^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$"
timeFormat: "%Y-%m-%dT%H:%M:%S.%L%z"
timeKey: time
- name: catchall
regex: "^(?<message>.*)$ }"
logfmt: []
json: []
env: []
podAnnotations: {}
fullConfigMap: false
existingConfigMap: ""
rawConfig: |-
@INCLUDE fluent-bit-service.conf
@INCLUDE fluent-bit-input.conf
@INCLUDE fluent-bit-filter.conf
@INCLUDE fluent-bit-output.conf
extraEntries:
input: ""
audit: ""
filter: |
Merge_Parser catchall
Keep_Log Off
output: ""
extraPorts: []
extraVolumes: []
extraVolumeMounts: []
resources: {}
hostNetwork: false
dnsPolicy: ClusterFirst
tolerations: []
nodeSelector: {}
affinity: {}
service:
flush: 1
logLevel: info
input:
tail:
memBufLimit: 5MB
parser: cri-mod
path: /var/log/containers/*.log
ignore_older: ""
systemd:
enabled: false
filters:
systemdUnit:
- docker.service
- kubelet.service
- node-problem-detector.service
maxEntries: 1000
readFromTail: true
stripUnderscores: false
tag: host.*
audit:
enable: false
input:
memBufLimit: 35MB
parser: docker
tag: audit.*
path: /var/log/kube-apiserver-audit.log
bufferChunkSize: 2MB
bufferMaxSize: 10MB
skipLongLines: true
key: kubernetes-audit
filter:
kubeURL: https://kubernetes.default.svc:443
kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kubeTag: kube
kubeTagPrefix: kube.var.log.containers.
mergeJSONLog: true
mergeLogKey: ""
enableParser: true
enableExclude: true
useJournal: false
rbac:
create: true
pspEnabled: false
taildb:
directory: /var/lib/fluent-bit
serviceAccount:
# Specifies whether a ServiceAccount should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
## Specifies security settings for a container
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
securityContext: {}
# securityContext:
# privileged: true
## Specifies security settings for a pod
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
podSecurityContext: {}
# podSecurityContext:
# runAsUser: 1000
2. Install the Fluent Bit helm chart, using values created in the previous step.
helm install --name fluent-bit -f values.yaml stable/fluent-bit
Summary
In this post, I shared the steps for deploying Fluentd and Fluent Bit in a Forwarding pattern.
In Part 4 I’ll wrap up with creating indices in Kibana and viewing the results.
Originally published at https://georgegoh.github.io.