Suganya G
Suganya G
Mar 16 · 3 min read

Application logs will take an important role to check errors and debug the code in the application. Centralized log management needs to be enabled to collect all your log data in one place for streamlined monitoring and better insight. We can use many tools like ELK,loggly, Graylog ext..

Customer scenario:

  • The customer had a logging server there rsyslog was enabled and all their user had access to this server to check the logs.
  • Every release new log will generate in the application server and it will integrate to the logging server.
  • We encourage the client to push all logs to cloud watch logs and create Cloudwatch log read-only IAM user to access the logs in aws console level.
  • We had used cloudwatch logs agent to push logs from logging to cloud watch. But Client faced the issue that whenever the new log generated he/she needs to update the config file with the new log file path.

Send logs via Fluentd to cloud watch logs:

We found Fluentd that supports *.log in directory basis (ex: /var/log/app/*/.log). It is easy to install and add plugins in few steps.

Let me explain, how we did this in customer infra,

  • Install Fluentd agent in the logging server

For ubuntu

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

For Redhat or centOS

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
  • Install cloudwatch logs fluentd plugin
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-cloudwatch-logs
  • I am going to use grok parser for filtering logs with formate. So I installed the grok parser plugin
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-grok-parser
  • Add the fluentd log config in the /etc/td-agent/td-agent.conf file (change the region as per the cloudwatch logs region)
<source>
@type tail
path /var/log/apps/*.log
exclude_path ["/var/log/apps/*.gz", "/var/log/apps/*.zip"]
pos_file /var/log/td-agent/apps.pos
tag apps-logs
<parse>
@type grok
grok_pattern %{CISCOTIMESTAMP:timestamp} %{URIHOST:host} %{GREEDYDATA:service} %{NUMBER}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}.%{SECOND} %{GREEDYDATA:message}
time_format "%b %d %H:%M:%S"
</parse>
</source>
<match apps-logs>
@type cloudwatch_logs
log_group_name staging-fluentd-apps-logs
log_stream_name apps-logs
auto_create_stream true
region us-east-1
</match>
  • Screenshot for Sample logs with grok pattern using grok debugger
Pattern the app log using Grok debugger

Add the EC2 role with cloudwatch logs access and add it to the EC2 instance

Now we can restart the td-agent service by running “service td-agent restart”.

Here we go!! Open the AWS console and go to cloud watch logs and verify the logs.

Filter the report service in Cloudwatch logs

That's it !!! finally, all logs will report to cloudwatch and Cloudwatch IAM users can view the logs no rework needed in fluentd config whenever new log added :) Hope you find this blog useful. Happy logging!!!

Searce Engineering

We identify better ways of doing things!

Suganya G

Written by

Suganya G

Associate Cloud Architect

Searce Engineering

We identify better ways of doing things!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade