Configuring Fluentbit on Kubernetes for AWS Elasticsearch

Bahubali Shetti
Sep 27, 2018 · 7 min read

Component basics:

Kubernetes Logging:

Fluentbit:

Elasticsearch:

Prerequisites

Application and Kubernetes logs in Elasticsearch

AWS Elasticsearch Cognito login with user/password
AWS Elasticsearch displaying application log
AWS Elasticsearch displaying system level log

Configuring and deploying fluentbit for AWS Elasticsearch

Setting up and configuring AWS Elasticsearch

Deploying fluentbit with es-proxy

$ kubectl create namespace logging
$ kubectl create -f fluent-bit-service-account.yaml
$ kubectl create -f fluent-bit-role.yaml
$ kubectl create -f fluent-bit-role-binding.yaml
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
Logstash_Format On
Retry_Limit False
tls Off <---- must be configured to Off (On is default)
tls.verify Off
kubectl create -f ./output/elasticsearch/fluent-bit-configmap.yaml
        - name: AWS_ACCESS_KEY_ID
value: "YOURAWSACCESSKEY"
- name: AWS_SECRET_ACCESS_KEY
value: "YOURAWSSECRETACCESSKEY"
- name: ES_ENDPOINT
value: "YOURESENDPOINT"
kubectl create -f ./output/es-proxy/es-proxy-deployment.yaml
kubectl create -f ./output/es-proxy/es-proxy-service.yaml
kubectl create -f ./output/elasticsearch/fluent-bit-ds-with-proxy.yaml

For more information on Cloud PKS (formerly VKE):

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade