How to send application logs to Cloudwatch using Fluentd

Suganya G
Suganya G
Mar 16 · 3 min read

Application logs will take an important role to check errors and debug the code in the application. Centralized log management needs to be enabled to collect all your log data in one place for streamlined monitoring and better insight. We can use many tools like ELK,loggly, Graylog ext..

Customer scenario:

  • The customer had a logging server there rsyslog was enabled and all their user had access to this server to check the logs.
  • Every release new log will generate in the application server and it will integrate to the logging server.
  • We encourage the client to push all logs to cloud watch logs and create Cloudwatch log read-only IAM user to access the logs in aws console level.
  • We had used cloudwatch logs agent to push logs from logging to cloud watch. But Client faced the issue that whenever the new log generated he/she needs to update the config file with the new log file path.

Send logs via Fluentd to cloud watch logs:

We found Fluentd that supports *.log in directory basis (ex: /var/log/app/*/.log). It is easy to install and add plugins in few steps.

Let me explain, how we did this in customer infra,

  • Install Fluentd agent in the logging server

For ubuntu

For Redhat or centOS

  • Install cloudwatch logs fluentd plugin
  • I am going to use grok parser for filtering logs with formate. So I installed the grok parser plugin
  • Add the fluentd log config in the /etc/td-agent/td-agent.conf file (change the region as per the cloudwatch logs region)
  • Screenshot for Sample logs with grok pattern using grok debugger

Add the EC2 role with cloudwatch logs access and add it to the EC2 instance

Now we can restart the td-agent service by running “service td-agent restart”.

Here we go!! Open the AWS console and go to cloud watch logs and verify the logs.

That's it !!! finally, all logs will report to cloudwatch and Cloudwatch IAM users can view the logs no rework needed in fluentd config whenever new log added :) Hope you find this blog useful. Happy logging!!!

Searce Engineering

We identify better ways of doing things!

Suganya G

Written by

Suganya G

Associate Solutions Architect

Searce Engineering

We identify better ways of doing things!