Sending container logs from AWS EKS to AWS CloudWatch using Fluentd
Centralized logging is one of the key aspects of today’s modern cloud-native architectures. We do not want our main microservices besides performing the business logic to do the heavy-lifting task of writing logs to a centralized location as well say, AWS CloudWatch, Elasticsearch, or any third-party HTTPS endpoints. As the best practices, these heavy-lifting tasks such as retry, logging, security (mTLS), etc should be decoupled from the main microservices.
In this blog, we will see, how we can leverage the Kubernetes DaemonSet object to run the Fluentd container in the AWS (Amazon Web Services) EKS (Elastic Kubernetes Service) cluster to collect all the logs from all the containers and then write to a centralized location, AWS Cloudwatch.
Please note that we cannot run a Kubernetes DaemonSet in AWS EKS with Fargate as daemon sets are still not supported yet on Fargate (Refer to the official AWS documentation link here). So if you still want to run Fluentd with EKS (Fargate) then you can run them as a sidecar pattern (multi-container Pod) and not as DaemonSet.
In this blog, we will see how you can configure and run Fluentd as a Kubernetes DaemonSet object in the AWS EKS cluster (with NodeGroup).
Our AWS logging Architecture:-
