Observability Services Part 1 : Fuentd-ElasticSearch-Kibana : Centralized Logging in Kubernetes

Observability Services for Microservice and Kubernetes

Prerequisite:

• Install the Docker Desktop
• Enable the Kubernetes in Docker Desktop

Note : Increased the RAM to 7or 8GB, CPUs to 4 and swap space to 3 or 4GB and restarted the Docker Desktop. kubectl works fine now.

Fluentd

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the Unified Logging Layer).

Fluentd to ElasticSearch

Fluentd is a Ruby-based open-source log collector and processor created in 2011. Fluentd uses about 40 MB of memory and can handle over 10,000 events per second. More than 500 different plugins are available. Fluentd is similar in operation to logstash on the elk stack.

Elasticsearch is a distributed, scalable, JSON-based search and analytics engine. It is popularly used as an elk stack (Elasticsearch, Kibana, Beats, and Logstash). Kibana is the visualizing tool for the Elasticsearch data.

Fluentd Docker

I would recommend to start with the official fluentd docker image.
You may want to build your own image if you want to install plugins. In this demo I will be using the fluentd elasticsearch plugin
It's pretty simple to adjust fluentd to send logs to any other destination in case you are not an elasticsearch user.

Let’s build our docker image in the introduction folder:

cd kubernetes
#note: use your own tag!
docker build . -t aimvector/fluentd-demo
#note: use your own tag!
docker push aimvector/fluentd-demo

https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes/dockerfiles

Fluentd Namespace

Let's create a fluentd namespace:

kubectl create ns fluentd

Fluentd Configmap

In my fluentd introduction video, I talk about how fluentd allows us to simplify our configs using the include statement.
This helps us prevent having a large complex file.

We have 5 files in our fluentd-configmap.yaml :

• fluent.conf: Our main config which includes all other configurations
• pods-kind-fluent.conf: tail config that sources all pod logs on the kind cluster. Note: kind cluster writes its log in a different format
• pods-fluent.conf: tail config that sources all pod logs on the kubernetes host in the cloud.
  Note: When running K8s in the cloud, logs may go into JSON format.
• file-fluent.conf: match config to capture all logs and write it to file for testing log collection
  Note: This is great to test if collection of logs works
• elastic-fluent.conf: match config that captures all logs and sends it to elasticseach

Let’s deploy our configmap:

kubectl apply -f fluentd-configmap.yaml

https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes/dockerfiles

Fluentd Daemonset

Let’s deploy our daemonset:

kubectl apply -f fluentd-rbac.yaml 
kubectl apply -f fluentd.yaml
kubectl -n fluentd get pods

Let’s deploy our example app that writes logs to stdout

kubectl apply -f counter.yaml
kubectl get pods

https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes

ElasticSearch and Kibana

kubectl create ns elastic-kibana
# deploy elastic search
kubectl -n elastic-kibana apply -f elastic-demo.yaml
kubectl -n elastic-kibana get pods
# deploy kibana
kubectl -n elastic-kibana apply -f kibana-demo.yaml
kubectl -n elastic-kibana get pods

Kubectl : Access the Logfile of Container

kubectl -n elastic-kibana get pods
kubectl logs pods/elasticsearch-6f6546d794-gqflb -n elastic-kibana

Access Kibana

kubectl -n elastic-kibana port-forward svc/kibana 5601

Check Kibana Dashboard : http://localhost:5601

Health Logs: http://localhost:5601/api/task_manager/_health

Access ElasticSearch

kubectl -n elastic-kibana port-forward svc/elasticsearch 9200

Check the ElasticSearch : http://localhost:9200/

Health Logs: http://localhost:9200/_cluster/health?pretty

Resource Files:

kubernetes/Fluentd-Kubernetes/elastic at main · dijin123/kubernetes

Reference :

docker-development-youtube-series/monitoring/logging/fluentd/kubernetes at master ·…