Monitor your AWS account to detect suspicious behavior in real time

You can track every change made to your AWS account with CloudTrail. Did you know that you can also monitor your AWS account in near real time with custom rules specific for your use case?

By combining CloudTrail, S3, SNS, and Lambda, you can run a piece of code to check the API activity in your account. Because of the reporting frequency of CloudTrail, this will happen approximately every 5 minutes. This post explains how to deploy a solution to monitor your EC2 instance tags for suspicious behavior.

The following figure shows how this works on a high level.

Monitor CloudTrail with Lambda

Let’s look at a concrete example.

What is suspicious behavior?

CloudTrail records a lot of API activity. Your job is to determine which activities are suspicious. Here are a few ideas:

• A security group was changed to open a port to the outside world (0.0.0.0/0).
• An IAM user was created outside of normal business hours.
• An EC2 instance was started without following your company’s tag schema (for example, you may tag technical ownership, cost ownership, and so on).

The example that follows implements the idea of EC2 instance tag monitoring.

Monitoring EC2 instance tags

Each time CloudTrail has new data for you, a Lambda function is triggered. The Lambda function needs to do the following:

1. Understand the input data generated from SNS.
2. Download the compressed CloudTrail files from an S3 bucket.
3. Uncompress the files.
4. Iterate through the API activities, looking for EC2 tag-related events: RunInstances, CreateTags, and DeleteTags.
5. Alert if the tag schema was violated.

Fortunately the code has already been written, so we won’t dive into Node.js code this time. Instead, we’ll focus on deploying this solution.

Deploying the solution

Lambda can be deployed almost entirely with CloudFormation. A few steps are required to prepare everything you need:

1. Choose an AWS region you want to monitor (referenced as $region in the following).
2. Create a SNS topic in $region, and subscribe to the topic via email. Alerts will be sent to this topic.
3. Download the code by running wget https://github.com/widdix/aws-tag-watch/archive/master.zip in your terminal.
4. Run unzip master.zip in your terminal.
5. Change dir by running cd aws-tag-watch-master/.
6. Run npm install in your terminal to install Node.js dependencies.
7. Edit config.json, and set region to $region and alertTopicArn to the ARN of your SNS topic from step 1.
8. Execute ./bundle.sh in your console.
9. Upload aws-tag-watch.zip to S3 (the bucket must be in $region).
10. Create a CloudFormation stack based on template.json.
11. Unfortunately, Lambda support in CloudFormation isn’t perfect, so you need to perform one permission task manually to allow your Lambda function to be invoked from SNS:
    # — function-name please fill in LambdaFunctionName output from CloudFormation stack
    # — source-arn please fill in TrailTopicArn output from CloudFormation stack
    $ aws lambda add-permission — function-name “…” — statement-id “s1” — action “lambda:invokeFunction” — principal “sns.amazonaws.com” — source-arn “…”

Now your AWS account in $region is monitored. Whenever you run a new EC2 instance or change the tags of an existing EC2 instance, the Lambda function will check whether you’re sticking to the tag schema.

Room for improvement

Raising an alert via email isn’t that helpful if you are working on a team. You may want to look at OpsGenie, which integrates nicely with SNS.

Read more about Amazon Web Services

Amazon Web Services in Action

Are you interested in learning more about Amazon Web Services? Andreas and I are writing a book called Amazon Web Services in Action, published by Manning. The book is written for developers and DevOps engineers who are moving traditionally deployed distributed applications to the AWS platform. No experience with AWS is required.