Watch Over Your AWS Castle

Building a Monitoring Dashboard

In AWS cloud, spinning up servers, databases, load balancers and the rest of 100+ services can be done in matter of minutes. It has never been easier to design powerful cloud architectures with seamless integration of many services. While enjoying these services, we must not forget that security is of paramount importance to any production level application. In terms of Security, AWS offers a shared responsibility model.

Shared Responsibility Model

AWS is responsible for the security of the physical infrastructure within their data-centers across the globe. An AWS customer is responsible for his application’s security. As shown in the above diagram, there are many areas that a customer should establish security for.

  1. User Data
  2. Access Management
  3. Network Security
  4. Data encryption at Client-Side, Server-Side and Network Traffic

Center for Information Security(CIS) has well defined recommendations to secure AWS application. This paper is called AWS CIS Foundations Benchmark and it is a must to read!

AWS provides different tools and services to tighten up the security of cloud applications. It’s the customer’s responsibility to utilize them and secure the above mentioned areas.

Continuous Monitoring

Continuous monitoring plays a great role in enforcing security of an AWS cloud application. Besides, there is a separate section for monitoring in AWS CIS Benchmark. Monitoring helps to detect cyberattacks and application anomalies and take counter actions instantaneously. So, how could we monitor our applications in AWS?

Following AWS services can be used to meet the needs to secure the environment.

  1. AWS CloudTrail (Monitor API calls to AWS infrastructure)
  2. AWS CloudWatch (Default logging service of AWS)
  3. VPC Flow Logs (Logs all the traffic to your VPC)
  4. AWS IAM (Identify And Access Management)
  5. AWS Config
  6. AWS X-Ray
  7. AWS WAF (Web Application Firewall)
  8. AWS Inspector

The rest of this post, will show you step by step instructions to build a near realtime dashboard to monitor AWS console users activities, Internet bot attacks, and Critical configuration changes within your AWS environment. We will use AWS IAM, AWS CloudTrail, AWS CloudWatch and VPC Flow Logs services to build the dashboard.

Building a Monitoring Dashboard

To show useful information on the dashboard, we have to setup different datasources. In this post, I will show how to setup CloudTrail and VPC Flow Logs to generate a dashboard using CloudWatch. We could setup other AWS services like AWS Config as datasources to CloudWatch.

Setting Up datasources for CloudWatch

Configure VPC Flow Logs

VPC flow logs detects all traffic to your VPC. It will send that logs to CloudWatch. As the first step, goto VPC section of your AWS console and select your VPC. Then select the Flow Logs tab and click ‘Create Flow Log’.

A new window will popup to configure flow logs for the selected VPC. Select ‘All’ traffic under Filter dropdown and select a role. If you don’t have a role setup, click on ‘Set Up Permission’ link and follow its wizard.

Create Flow Log Window
Creating a new role

Now that you have created a new role, go back to the Create Flow Log window and fill in the details. Select the role name and provide a log group name. eg: VPC/FlowLogs.

Setting the created role

Click on ‘Create Flow Log’ button. You will see the newly created flow log under the Flow Log tab.

Setup a CloudWatch Metric Filter

Once the VPC flow log is created, it sends logs to CloudWatch under the given name. (VPC/FlowLogs). We need to create a Metric Filter with that log group.

Go to AWS CloudWatch service. Search the log group name i.e. VPC/FlowLogs.

When you click on the log stream, it will show all the Flow Logs for your VPC and for other resources that reside inside your VPC. Following picture shows logs of an EC2 instance. This EC2 instance’s security group allows port 80 ingress traffic from Anywhere. ‘REJECT OK’ logs are from rejected requests(Probably bot attacks) and ‘ACCEPT OK’ logs are from accepted requests.

Let’s create a Metric Filter for REJECTED requests in VPC flow logs. Then we can create a widget from that Metric and display it on our dashboard.

Select the VPC/FlowLogs log group and click on ‘Create Metric Filter’. As the Filter Pattern provide ‘REJECT OK’. Click on ‘Assign Metric’ button.

Give a filter name and click on Create Filter to create a filter for REJECTED Requests.

It will probably take about 5–10 minutes to create the filter. After 5–10 minutes click on the Metric Link. (LogMetrics/VPC_Rejected_Requests)

Select ‘LogMetrics’ under Custom Namespaces. Go to the ‘Graphed Metrics’ tab and select ‘Sum’ for statistics.

This chart shows the rejected traffic from VPC with time. Now it is ready to add as a widget to the dashboard. Before adding this widget, let’s create few more widgets as well.

CloudTrail to Monitor Resource API Calls

If your AWS account is used by many developers, there could be situation that one of the ex-developers shuts down resources intentionally or unintentionally. To trace such activities, we must use AWS CloudTrail.

Click on ‘Get Started Now’ and give your CloudTrail a name. You can setup CloudTrail either for all the regions or the current region.

You can setup S3 buckets to store logs from CloudTrail. CloudTrail will deliver logs to S3 in every 5minutes. You can setup S3 lifecycle policies to put logs to infrequent access or archive in Glacier such that never delete log data. It is important to preserve logs in case of audit requirements.

Configure CloudTrail with CloudWatch

Now that we have created our CloudTrail, it is time to configure it to send logs to CloudWatch. Go to the created CloudTrail and expand CloudWatch Logs section. Hit ‘Configure’.

As we did in VPC flow logs configurations, we need to give a log group name in CloudWatch where all the CloudTrails data will be sent over to. I kept the default name i.e. CloudTrail/DefaultGroup

At the next step, you need to create a new IAM role and provide CloudWatchLogsFullAccess policy. Then assign it to your CloudTrail to allow logging permission to CloudWatch.

After few minutes, goto CloudWatch and search for the group name ‘CloudTrail/DefaultGroup’. You will see the logs from CloudTrail.

Now its time to create different metric filters to filter information from CloudTrail logs. Select log group and click on ‘Create Metric Filter’ action.

In the ‘Define Logs Metric Filter’ window, you must define the filter pattern. I wanted to console activities of each my AWS account users. The filter condition is shown on the diagram below. I provided my name to log my activities.

Click on ‘Assign Metric’ and goto the next window, where it will ask name of your Metric.

In about 5–10 minutes after creating the metric, select the metric name on the list. Navigate to Graphed Metrics tab and select ‘SUM’ under statistics.

Now click on ‘Add to dashboard’ action on the metrics windows. If you don’t already have a new dashboard, create new one or select any type of dashboard.

Similarly add filters for other users within the AWS account. You can also add content filters to any service that is being used in your environment. Finally, you will have a much more sophisticated dashboard.

Following is an example filter for Security Groups. It will detect creation of new security groups, updating security groups, deleting security groups etc…

{($.eventName = AuthorizeSecurityGroupIngress) ||
($.eventName = AuthorizeSecurityGroupEgress) ||
($.eventName = RevokeSecurityGroupIngress) ||
($.eventName = RevokeSecurityGroupEgress) ||
($.eventName = CreateSecurityGroup) ||
($.eventName = DeleteSecurityGroup) }

Following picture shows a sample dashboard that is configured with VPC Flow Logs and CloudTrail.

Show your support

Clapping shows how much you appreciated Manoj Fernando’s story.