Log collection with Amazon AWS CloudWatch

When something goes wrong you normally need to look at the log files to figure out what has happened. But getting access to the machine to look at the files is often difficult and what you really want is a central place were all your logs are saved and you can search/ aggregate them. This is also really useful if you need to do some BI stuff on them as you can just use them for a hadoop job etc…

For ages now we have just been shipping files around and not really used any professional solution for this. While there are loads out there they are all quite “big” and cumbersome to set up and maintain. As my new goal in life is to spend near to no time doing sys admin tasks I was looking for something really easy with which I can just dump my logs in some central place.

Welcome to http://aws.amazon.com/de/cloudwatch/

While saving log files is not the main objective of this AWS product, it does quite a good job at just saving away everything I will ever need.

Because it is fairly cumbersome to write the config here a copy paste of the awslogs.conf. This will pretty much save away all the important files for a standart webserver using apache/ httpd.

[general]
# Path to the CloudWatch Logs agent's state file. The agent uses this file to maintain
# client side state across its executions.
state_file = /var/lib/awslogs/agent-state


[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/messages

[/var/log/httpd/access_log]
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/httpd/access_log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/httpd/access_log

[/var/log/httpd/error_log]
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/httpd/error_log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/httpd/error_log

[/var/log/httpd/ssl_access_log]
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/httpd/ssl_access_log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/httpd/ssl_access_log

[/var/log/httpd/ssl_error_log]
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/httpd/ssl_error_log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/httpd/ssl_error_log

[/var/log/httpd/ssl_request_log]
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/httpd/ssl_request_log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/httpd/ssl_request_log

[/var/log/secure]
datetime_format = %b %d %H:%M:%S
file = /var/log/secure
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/secure

[/var/log/yum.log]
datetime_format = %b %d %H:%M:%S
file = /var/log/yum.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/yum.log

[/var/log/cron]
datetime_format = %b %d %H:%M:%S
file = /var/log/cron
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/cron
One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.