Managing Cloud Compliance at Scale with AWS Security Hub and Cloud Custodian

Photo by Nathanial Dahan on Unsplash

A well-architected multi-account strategy is the foundation of any enterprise-scale AWS environment. Taking a multi-account approach enables an organization to isolate AWS resources utilized by separate product teams and developers, while also making it easy to consolidate and centrally manage core functionality (billing, security, logging, etc.)

Presented within this highly informative session from re:Invent 2018, the structure shown in Image 1 illustrates an example enterprise-scale account strategy that allows the effective separation of AWS resources.

Image 1: Example AWS Multi-Account Strategy

While using multiple AWS accounts provides the highest level of resource isolation and consolidated management of key business functions, it also introduces the challenge of managing security and compliance at scale. With large organizations potentially operating thousands of AWS accounts, failure to effectively manage security and compliance across them can be particularly dangerous.

Security and Compliance in AWS: The Shared Responsibility Model

As you may know, security and compliance within AWS are addressed via a Shared Responsibility Model, where both AWS and the customer assume joint responsibility for meeting all requirements. Image 2 illustrates the Shared Responsibility Model, highlighting specific responsibilities for both AWS (orange) and their customers (blue).

Image 2: The AWS Shared Responsibility Model

In this model, AWS is responsible for the security ‘of’ the cloud, protecting the underlying infrastructure (hardware, software, networking, facilities) that runs all AWS services. Customers are responsible for security ‘in’ the cloud, tasked with configuring their chosen AWS Cloud services in a way that enables them to meet their specific security and compliance goals.

Failure within an organization to manage security and compliance across multiple AWS accounts can lead to potential disaster. To give some examples, poor security group rule management, publicly available S3 buckets or a lack of encryption enforcement can all leave an organization at risk of being compromised. All too often we read about established, cloud-hosted organizations suffering from data breaches resulting in Personally Identifiable Information (PII), Intellectual Property (IP), or trade secrets being leaked online. Such events almost always occur as a result of human error in the application of security or compliance protocols to cloud services. With the potential for thousands of individuals using AWS accounts to be making unsafe changes, ensuring a comprehensive, automated strategy for account security and compliance enforcement should be a priority for any AWS customer.

Examine the Big Picture: Introducing AWS Security Hub

AWS contains a variety of services designed to help organizations meet certain security and compliance goals, including:

• Amazon GuardDuty — threat detection that monitors for malicious activity and unauthorized behavior
• Amazon Macie — discover, classify and protect sensitive data
• Amazon Inspector — assess applications for exposure, vulnerabilities, and deviations from best practices

Although each provides distinct security assessment and detection capabilities, additional effort is required to consolidate and prioritize findings from GuardDuty, Macie and Inspector due to their separation within the AWS platform. To address this, AWS Security Hub was announced at re:Invent 2018 to provide a comprehensive view of high-priority security and compliance alerts from AWS services and AWS Partner Solutions. Results from GuardDuty, Macie and Inspector are consolidated within AWS Security Hub per AWS Region and reported using a standardized findings format. An added benefit, findings from AWS Partner Solutions (full list of AWS Partner Integrations available here) are also reported using this format.

While GuardDuty, Macie, and Inspector provide valuable assessment and detection capabilities, their use alone is by no means sufficient in meeting security and compliance needs. In addition to findings reported from these services and AWS Partner Integrations, the results of continuous configuration and compliance checks based on industry standards, such as the CIS AWS Foundations Benchmarks, can also be reported within Security Hub. Enforced via a set of AWS Config rules, the CIS AWS Foundations Benchmarks provide prescriptive guidance on how to configure a handful of AWS Services to meet various security and compliance best practices. The particular AWS Services covered by the guidelines are:

• AWS IAM
• AWS Config
• AWS CloudTrail
• AWS CloudWatch
• AWS SNS
• AWS S3
• AWS VPC

With AWS Security Hub and AWS Config enabled in each desired AWS Region, configuration and compliance checks associated with the CIS AWS Foundational Benchmarks are automatically added as rules within Config. Results from all security and compliance checks feeding into AWS Security Hub are collected on a single ‘Findings’ tab, as shown in Image 3.

Image 3: Security Hub Findings Page

Clicking on a specific finding allows you to collect more detailed information, including affected AWS Account, resource information and a link to documentation outlining a remediation strategy (see Image 4).

Image 4: Gathering Details on a Specific Finding in AWS Security Hub

Examining another key feature of Security Hub, individual findings can be filtered and grouped to create ‘Insights’. While 30 predefined, AWS managed insights are configured by default, custom insights can be created by using an aggregation statement and applying filters to the list of findings as required. Effective use of insights allows an organization to identify emerging trends or possible issues within their AWS environment. Image 5 illustrates a simple custom insight created to track compliance checks of Severity between 0 and 3 in a ‘FAILED’ state by AWS account over time.

Image 5: Example of a custom Security Hub Insight

At enterprise-scale, AWS Security Hub can be utilized within a centralized security account to monitor security and compliance across up to 1000 accounts. This is achieved using a master/member model, where the master Security Hub extends invitations for member accounts to accept. All Security Hub findings from accepted member accounts feed into Security Hub in the master account, providing a centralized view across an organization. Image 6 illustrates the member accounts associated with Security Hub within my centralized security AWS account.

Image 6: Security Hub Member Accounts

An important point to note, AWS Security Hub is a regional AWS service. This means that any steps required to create master/member relationships must be executed within each AWS Region applicable to your organization. In an attempt to try and make this configuration easier, AWS Labs have published a script on GitHub that executes steps to enable Security Hub per-region and configure the master/member relationship within. While the script does remove a lot of the setup burden, the configuration process is definitely a little convoluted and something I’m confident AWS will look to address in subsequent releases of the Security Hub service.

In addition to providing a consolidated view of security and compliance across an organization’s AWS environment, AWS Security Hub also provides the ability to automate the response to discovered findings or insights. This response can come in many forms, from sending informative messages to chat, email or ticketing systems all the way to utilizing automated remediation workflows triggered by CloudWatch Events and Lambda functions. The latter is made possible by the fact that Security Hub publishes all events to CloudWatch Events. As an example, the following JSON block is a CloudWatch Event Pattern to catch Security Hub findings that currently have a status of ‘FAILED’.

{   
  "source": [
     "aws.securityhub"
  ],
  "detail-type": [
     "Security Hub Findings - Imported"
  ],
  "detail": {
    "findings": {
      "Compliance": {
        "Status": [
          "FAILED"
        ]
      }
    }
  }
}

The JSON within the code block is a basic example that can be enhanced with additional finding or insight specific key-value pairs to catch unique events. With this added level of detail, any CloudWatch Event Target can be used to remediate the issue, such as a Lambda function, Step Function or SSM Run command.

Extending Your Reach: Integrating AWS Security Hub with Cloud Custodian

AWS Security Hub currently provides the ability to detect and react to a variety of security and compliance-related issues raised by Amazon GuardDuty, Macie, Inspector, Partner Solutions and the CIS AWS Frameworks Benchmarks. However, in many cases organizations have additional security or compliance requirements they are looking to address that extend beyond these capabilities. Enter Cloud Custodian.

A cloud-agnostic, open-source rules-engine for managing and remediating security, compliance and governance issues, Cloud Custodian employs user-created YAML configuration policies to provide security and compliance as code. These configuration policies support over 140 AWS resource types and often contain the following key items:

• name — Unique name identifier for the policy
• resource — The AWS Resource type the policy is written for
• mode — The Cloud Custodian mode to run this specific policy in. Some commonly employed options are ‘pull’, ‘cloudtrail’ or ‘periodic’. A full list of available options and their behavior can be found here
• filters — Provides the ability to query and isolate specific AWS resources
• actions —Defines the actions to apply to resources identified by the filter

Cloud Custodian was originally developed by Capital One and is employed by many large organizations to manage and remediate security and compliance issues across their AWS environments. However, a common complaint with Cloud Custodian since its creation has been the lack of a consolidated view detailing results from applied configuration policies. As a major step towards addressing this shortcoming, AWS announced the integration of Cloud Custodian as a findings provider within AWS Security Hub at the end of 2018. With this capability, compliance and security issues identified via Cloud Custodian policies can be published to Security Hub and searched or filtered in the same way as any other findings.

To give an example, Image 7 illustrates a Cloud Custodian Policy that reports the existence of any unencrypted EBS volumes to AWS Security Hub.

Mapping to the key sections identified above:

• Line 2: Provides a unique name for the policy
• Line 5: Identifies the AWS Resource Types as EBS volume
• Line 6: The mode is set to ‘periodic’, which will create a scheduled CloudWatch Event Rule that triggers a Lambda function every 15 minutes
• Line 10: The filters specify that we would like to find EBS volumes with the Encrypted value set to false
• Line 15: The actions specify that we would like to use the ‘post-finding’ action, which publishes results to AWS Security Hub

Image 7: Example Cloud Custodian Policy to detect unencrypted EBS volumes

Steps detailing the installation of Cloud Custodian are beyond the scope of this post, but detailed instructions on how to do so can be found here. As a best practice, organizations should look to utilize dedicated EC2 instances or containers with applicable IAM roles to execute Cloud Custodian commands.

With Cloud Custodian installed, the execution of a Cloud Custodian policy against a single account is done using the ‘run’ command. Detailed usage information for this command can be found via:

custodian run -h

To execute the policy from Image 7 against my security AWS account, I enter the following on my command line (substituting in a correct value for account_id):

custodian run -s output ebs-notify-unencrypted-volume-periodic.yml  --assume arn:aws:iam::<account_id>:role/custodian_access_role

which outputs the following two lines indicating successful deployment:

custodian.policy:INFO Provisioning policy lambda notify-unencrypted-ebs-periodic
custodian.serverless:INFO Publishing custodian policy lambda function custodian-notify-unencrypted-ebs-periodic

Looking in my AWS account, I see a CloudWatch Event Rule (Image 8) and corresponding Lambda function (Image 9) have been created by Cloud Custodian to represent and enforce my policy:

Image 8: CloudWatch Event created to trigger Lambda Function

Image 9: Lambda Function Representing the Cloud Custodian Policy

To showcase Cloud Custodian’s integration with Security Hub, I created two unencrypted EBS volumes within my AWS account. After waiting a few minutes for the CloudWatch Event to trigger the Lambda function, I was able to see two new finding results within AWS Security Hub (shown in Image 10)

Image 10: Two Findings in Security Hub indicating the presence of unencrypted EBS volumes

This example offers a simple demonstration of Cloud Custodian and its ability to integrate with AWS Security Hub. However, it should be noted that the capabilities of Cloud Custodian extend far beyond centralized reporting. For example, further actions could have been specified to email the creator of the EBS volume or auto-terminate the offending items. In short, Cloud Custodian is a powerful tool enabling the formulation of highly customized configuration policies and advanced remediation strategies.

To meet security and compliance goals within enterprise-scale AWS environments, Cloud Custodian provides the capability to simultaneously deploy configuration policies in multiple regions of all accounts within an organizations AWS environment. This is made possible through the use of Cloud Custodian’s c7n-org utility in conjunction with effective cross-account IAM roles.

To identify AWS accounts to apply policies against, Cloud Custodian relies on the existence of a YAML config file with specific account information. The c7n-org utility page provides an example of the required structure for those who wish to self-author the file. Alternatively, scripts have been published which auto-generate it using information gathered via the AWS Organizations API.

In addition to the YAML config file containing AWS account information, c7n-org depends on cross-account IAM role usage within an organizations AWS environment. Although specific permissions required within IAM Policies depends on the actions defined within configuration policies, baseline permissions required for deployment are outlined here. To successfully execute in a multi-account scenario, Cloud Custodian should be run using an IAM role that has the ability to assume IAM roles within target AWS accounts, as illustrated in Image 11:

Image 11: Illustration of IAM Role Usage in Multi-Account Configuration with c7n-org utility

With IAM roles in place and account information available within a config file (let’s call it accounts.yml), the Cloud Custodian policy from Image 7 can be applied to all AWS accounts within an organization by executing:

c7n-org run -c accounts.yml -s output -u ebs-notify-unencrypted-volume-periodic.yml

which outputs the following to indicate application against specified regions within multiple AWS accounts:

c7n_org:INFO Ran account:bcutler-aws-sharedservices region:us-east-1 policy:notify-unencrypted-ebs-periodic matched:16 time:19.41
c7n_org:INFO Ran account:bcutler-aws-networkinfrastructure region:us-east-1 policy:notify-unencrypted-ebs-periodic matched:16 time:20.31
c7n_org:INFO Ran account:bcutler-aws-sharedservices region:us-west-2 policy:notify-unencrypted-ebs-periodic matched:16 time:33.12
c7n_org:INFO Ran account:bcutler-aws-networkinfrastructure region:us-west-2 policy:notify-unencrypted-ebs-periodic matched:16 time:36.01
c7n_org:INFO Policy resource counts Counter({'notify-unencrypted-ebs-periodic': 64})

With the regional enablement of AWS Security Hub in target accounts and their status as Security Hub members, all results generated from the deployed Cloud Custodian policy are reported as findings within the centralized AWS Security Hub in my security account.

Although a brief introduction, the above example demonstrates how Cloud Custodian can be utilized by organizations to identify, report and take action on cloud security and governance issues at scale across their AWS environment

Putting It All Together

Utilizing multiple AWS accounts is an essential part of helping enterprise-scale organizations to be successful in the cloud. Doing so enables the centralized management of key business functionality and segregation of AWS resources used by disparate product teams or developers. However, it also creates an issue around maintaining security and compliance at scale.

AWS Security Hub was released in an attempt to provide organizations with a centralized view of how well they are meeting their specific security and compliance goals. The ability to view findings from native AWS integrations, Partner Solutions, and CIS AWS Frameworks Benchmarks in a single location is hugely beneficial.

However, organizations will often have certain security and compliance needs that extend beyond the findings available from these sources. Utilizing a popular open-source tool, Cloud Custodian, enables organizations to define highly-specific security, compliance and governance policies as code. The recently announced integration between AWS Security Hub and Cloud Custodian has added tremendous value, providing Cloud Custodian users with a comprehensive, visual insight into results being generated by configuration policies.

By utilizing a combination of the tools discussed, the challenge of managing security and compliance at scale in the cloud can be addressed effectively, reducing the risk of future compromise across the business.