Export Cloudwatch Log To S3

Jim Kang
Jim Kang
Feb 21 · 2 min read
Photo by george0131 (CC By 2.0)

Don’t tell me that you will actually go through each one of the CloudWatch logs and troubleshoot, it’s just not… civil !

You’ll not actually click through each of them and search, right? So we’ll need to export them to S3 for better search, like put all in a folder and have something like VSCode to search within each file.

Setup S3 and its permission

Create a S3 bucket in your AWS account. When it comes to bucket policy, use this:

{
"Version": "2008-10-17",
"Id": "Policy1335892530063",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.YOUR-REGION.amazonaws.com"
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
},
{
"Effect": "Allow",
"Principal": {
"Service": "logs.YOUR-REGION.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}

REMEMBER TO CHANGE YOUR-REGION AND YOUR-BUCKET-NAME !!!

Export Those Logs

Make sure to choose the bucket that you just created. It could take a bit of time but it’ll appear in your S3 bucket with a folder called exportedlogs if you leave the setting unchanged (above picture)

You could use Cyberduck or Transmit if you’re using a mac. Or feel free to use the command line.

Useful Commands

After you download your log, you’ll see that there are a bunch of folders, and each folder contains a file called 00000.gz So these commands should help you out!

Pull Files Out From Subdirectories

shell script code

Remove All Empty Folders

find . -mindepth 1 -maxdepth 1 -type d -exec rm -r {} \;

File Count In The Current Directory

so you are sure the numbers is as expected

ls | wc -l

Unzip All .tar In The Current Folder

for a in ls -1 *.gz; do gzip -d $a; done

Now you can put all these files in a folder and search them in light speed!
Safe travel!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade