Observability Services Part 1 : Fuentd-ElasticSearch-Kibana : Centralized Logging in Kubernetes
Observability Services for Microservice and Kubernetes
Prerequisite:
- Install the Docker Desktop
- Enable the Kubernetes in Docker Desktop
Note : Increased the RAM to 7or 8GB, CPUs to 4 and swap space to 3 or 4GB and restarted the Docker Desktop. kubectl works fine now.
Fluentd
Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the Unified Logging Layer).
Fluentd to ElasticSearch
Fluentd is a Ruby-based open-source log collector and processor created in 2011. Fluentd uses about 40 MB of memory and can handle over 10,000 events per second. More than 500 different plugins are available. Fluentd is similar in operation to logstash on the elk stack.
Elasticsearch is a distributed, scalable, JSON-based search and analytics engine. It is popularly used as an elk stack (Elasticsearch, Kibana, Beats, and Logstash). Kibana is the visualizing tool for the Elasticsearch data.
Fluentd Docker
I would recommend to start with the official fluentd docker image.
You may want to build your own image if you want to install plugins. In this demo I will be using the fluentd
elasticsearch plugin
It's pretty simple to adjust fluentd
to send logs to any other destination in case you are not an elasticsearch
user.
Let’s build our docker image in the introduction folder:
cd kubernetes#note: use your own tag!
docker build . -t aimvector/fluentd-demo#note: use your own tag!
docker push aimvector/fluentd-demo
https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes/dockerfiles
Fluentd Namespace
Let's create a fluentd
namespace:
kubectl create ns fluentd
Fluentd Configmap
In my fluentd introduction video, I talk about how fluentd
allows us to simplify our configs using the include
statement.
This helps us prevent having a large complex file.
We have 5 files in our fluentd-configmap.yaml
:
- fluent.conf: Our main config which includes all other configurations
- pods-kind-fluent.conf:
tail
config that sources all pod logs on thekind
cluster. Note:kind
cluster writes its log in a different format - pods-fluent.conf:
tail
config that sources all pod logs on thekubernetes
host in the cloud.
Note: When running K8s in the cloud, logs may go into JSON format. - file-fluent.conf:
match
config to capture all logs and write it to file for testing log collection
Note: This is great to test if collection of logs works - elastic-fluent.conf:
match
config that captures all logs and sends it toelasticseach
Let’s deploy our configmap
:
kubectl apply -f fluentd-configmap.yaml
https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes/dockerfiles
Fluentd Daemonset
Let’s deploy our daemonset
:
kubectl apply -f fluentd-rbac.yaml
kubectl apply -f fluentd.yaml
kubectl -n fluentd get pods
Let’s deploy our example app that writes logs to stdout
kubectl apply -f counter.yaml
kubectl get pods
https://github.com/dijin123/kubernetes/tree/main/Fluentd-Kubernetes
ElasticSearch and Kibana
kubectl create ns elastic-kibana# deploy elastic search
kubectl -n elastic-kibana apply -f elastic-demo.yaml
kubectl -n elastic-kibana get pods# deploy kibana
kubectl -n elastic-kibana apply -f kibana-demo.yaml
kubectl -n elastic-kibana get pods
Kubectl : Access the Logfile of Container
kubectl -n elastic-kibana get pods
kubectl logs pods/elasticsearch-6f6546d794-gqflb -n elastic-kibana
Access Kibana
kubectl -n elastic-kibana port-forward svc/kibana 5601
Check Kibana Dashboard : http://localhost:5601
Health Logs: http://localhost:5601/api/task_manager/_health
Access ElasticSearch
kubectl -n elastic-kibana port-forward svc/elasticsearch 9200
Check the ElasticSearch : http://localhost:9200/
Health Logs: http://localhost:9200/_cluster/health?pretty