Searching CloudTrail Logs Easily with Amazon CloudSearch

AWS Startups
Oct 21, 2014 · 6 min read

In addition to the flexibility AWS provides startups in creating and deleting application infrastructure, AWS CloudTrail provides a key security service with its detailed security audit logs. It records the API calls made in your AWS account and publishes the resulting log files to an Amazon S3 bucket in JSON format. But without appropriate tooling, audit log review can be cumbersome. In today’s post, you’ll see how to set up a simple CloudTrail log analysis solution based on Amazon CloudSearch, a fully managed cloud-based search service.

I. Overall Architecture

Image for post
Image for post

II. Set Up AWS CloudTrail

In the CloudTrail console, click Advanced and create an SNS notification. For SNS topic (new), type something like “CloudTrail-notification.” Make sure the SNS notification for every log file delivery? is set to Yes.

Note: if you are using CloudTrail in multiple AWS regions, you should create one SNS topic per region.

Image for post
Image for post

III. Create the AWS CloudSearch Domain

Creating a script simplifies performing future deployments and changes. You can download and install the AWS CLI by following the steps in the AWS Command Line Interface User Guide.

The CloudSearch domain creation will take several minutes to complete. Download the domain creation script here.

The script takes care of the domain creation and is configured with a default domain name “cloudtrail-1” and created in the “us-east-1” region. You can easily customize the script to change domain name or AWS region. You can use a single CloudSearch domain to index CloudTrail logs from multiple regions.

IV. Create an SQS Queue

Creating a new SQS queue is easy with the AWS Management Console. In the SQS console, click Create New Queue and specify the following parameters:
• Queue Name: CloudTrail-sqs
• Default Visibility Timeout: 1 minute
• Message Retention Period: 14 days (maximum)
• Receive Message Wait Time: 20 seconds

Click Create Queue.

Image for post
Image for post

Next, click Queue Actions and click Subscribe Queue to an SNS Topic. For Choose a Topic, select the SNS topic that you created in the CloudTrail console. Click Subscribe. The AWS Console will automatically set up the required security policies.

Note: If you are using CloudTrail in multiple AWS regions, you should subscribe this SQS queue to each SNS topic in each region.

Image for post
Image for post

After several minutes you should see messages starting to arrive into your SQS queue.

V. Launch the Elastic Beanstalk Application

As the application needs to issue AWS API calls to Amazon S3 and to Amazon CloudSearch, we will also use an use an IAM role for EC2 to allow the application to make secure API requests from your instances without requiring you to manage the security credentials that the application uses. Let’s first create the required role in the console.

Navigate to the IAM console, click Roles in the navigation pane and then click Create New Role. Enter the role name, such as “cloudsearch-index,” and click Next Step. Then click Select for Amazon EC2 under AWS Services Roles. On the Set Permissions page, scroll down and click Custom Policy and Select. Copy and paste the policy below and give it a name. Click Next Step and then Create Role. This results in a policy like the following:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "cloudtrailworkerrole",
"Effect": "Allow",
"Action": [
"cloudsearch:DescribeDomains",
"cloudsearch:ListDomainNames",
"cloudsearch:document",
"s3:GetObject",
"s3:ListBucket",
"sqs:ChangeMessageVisibility",
"sqs:DeleteMessage",
"sqs:ReceiveMessage",
"cloudwatch:PutMetricData"
],
"Resource": [
"*"
]
}
]
}

Next you need to use the console to launch the Elastic Beanstalk application. You can download the .zip file here. The file .ebextensions/cloudtrail.config contains the CloudSearch domain name and region. You can change this file or later change PARAM1 and PARAM2 directly within Elastic Beanstalk.

option_settings:
"aws:elasticbeanstalk:application:environment":
PARAM1: cloudtrail-1
PARAM2: us-east-1

It is best to deploy your Beanstalk app in a VPC to get all the benefits of t2.micro instances: lowest-cost general purpose instance type with burstable CPU.

When you’re ready, go to the Elastic Beanstalk console and click Create a New Application. Deploying the Elastic Beanstalk is straightforward. Here are the required nondefault parameters:
• Environment tier: Worker
• Predefined configuration: Python
• Environment type: Load balancing, autoscaling (actually no load balancer is created with worker role; this option only creates the autoscaling group)

Click Next. Click Browse and upload the application .zip file you downloaded previously.

Click Next. Give a name to the Environment. The default is “cloudtrail1-env.”

Click Next. Select Create this environment inside a VPC.

Click Next. In the Configuration Details page, use these settings:

• Instance type: t2.micro
• Application health check URL: /
• Instance profile: cloudsearch-index

Click Next twice. For Worker Details, use these settings:

• Worker queue: CloudTrail-sqs
• HTTP path: /sns/
• MIME Type: keep “application/json”

Click Next. For VPC security group, you can use the default. (Actually you will not need any ingress network traffic.)

After a couple of minutes, the application status turns green, and your CloudSearch domain starts to be populated.

VI. Using CloudSearch

Here are some sample search requests to try:
matchall (this would view all documents, useful to explore facets)
event_source:’sqs.amazonaws.com’ (this would show all Amazon SQS events)
event_time:[‘2014–09–01T00:00:00Z’,’2014–09–02T00:00:00Z’] (this would show all events for Sept 1st 2014)

Note that CloudSearch displays facets on the right column, which helps you easily explore the data.

Image for post
Image for post

VII. Conclusion

AWS Startup Collection

For startups building on AWS.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store