Photo by Jessica To’oto’o on Unsplash

Custom AWS Config Rules

This article will attempt to explain how to create a custom AWS Config Rule. A working example is provided, using SAM and a Go-flavoured Lambda function.

All organisations, regardless of size, will have some sort of Policy. In large enterprises it will be formalised, codified and audited. In small startups it’s more likely to be yelled across the office. But it’s the same thing — a set of rules or idioms that must be followed. Often, Policy is associated with security, but that needn’t always be the case: a naming standard is an example of Policy.

Enforcement of Policy in AWS is achieved through correct configuration of resources (e.g. security groups or bucket policies). Infrastructure as Code can help here (think Terraform or CloudFormation) along with good practices such as code reviews or deploy-time checks.

However ongoing adherence, or compliance, to Policy is another thing entirely.

Let’s take an example: you have made the sensible decision not to expose SSH or RDP ports to the internet at large and your well-drilled CloudFormation templates reflect this. Consider the hapless engineer who temporarily opens up port 22 so they can SSH in from home and who, being hapless, forgets to revert the configuration afterwards. Or worse, some malignant manages to get their hands on a set of access keys and immediately cracks open an RDP port.

This is where AWS Config Rules come in. They provide an AWS-native way to perform dynamic compliance checking. In the above example they could immediately inform us that a security group is in breach of Policy.

There are many pre-rolled Managed Rules to cover such no-brainers as ‘vpc-flow-logs-enabled’ or ‘access-keys-rotated’. These are extremely useful. But sometimes just a little too rigid. You might, for instance, wish to exclude certain resources from being reported (again taking the example above, you might allow port 22 to be open on certain bastion hosts) or just require a policy more specific to your business.

Happily, Config Rule checks, even managed ones, are just Lambda functions. It’s entirely possible to write a custom Lambda function and wire it up as a Custom Config Rule. The rest of this article is dedicated to explaining how to do just that.

Custom Config Rules

Custom AWS Config Rules come in two flavours:

• Periodic: a check is run on a user-defined schedule to confirm compliance of existing resources. It probably involves enumerating some list of resources and checking each in turn.
• Triggered: a check is run whenever a resource of the specified type is created, or undergoes a configuration change.

We’ll focus on the Triggered checks here as they are more powerful and, frankly, more interesting.

The basic premise of a custom check is that it must make an evaluation as to whether a resource conforms to some policy. The result must be one of:

• COMPLIANT
• NON_COMPLIANT
• NON_APPLICABLE

The function must then make a call to the AWS API PutEvaluation endpoint with that result.

A Contrived Example

Let’s say that we have a Policy:

All Production Database EC2 Instances must have EBS Optimization enabled.

Yes it’s a little contrived, maybe ill-advised (why run your own database?), but still altogether plausible.

We can translate this into cloud-speak thusly:

If an EC2 instance has the tags ‘ENVIRONMENT=Production’ and ‘SERVICE=database’ then it must have EBS optimization enabled.

Naturally this assumes that the EC2 tags have been set correctly in the first place but, hey, we have to take some responsibility. (Or not: we could conceivably have a different Config rule that checks for the existence of these tags.

Anatomy of a Triggered event

Before we can passably attempt to explain any code, we need to delve into the data structure that AWS Config uses to communicate a triggered Event. This data structure, or ‘Event’, is passed into a specified Lambda handler whenever a change happens to any resource of the specified type. In our case that means any change to any EC2 instance.

Full documentation can be found here, but essentially an event looks like this:

{ 
    "invokingEvent": "...",
    "ruleParameters": "{\"myParameterKey\":\"myParameterValue\"}",
    "resultToken": "myResultToken",
    "eventLeftScope": false,
    "executionRoleArn": "arn:aws:iam::123456789012:role/config-role",
    "configRuleArn": "arn:aws:config:us-east-2:123456789012:config-rule/config-rule-0123456",
    "configRuleName": "change-triggered-config-rule",
    "configRuleId": "config-rule-0123456",
    "accountId": "123456789012",
    "version": "1.0"
}

Some points:

• We’ll need resultToken when we eventually make the call back to PutEvaluation.
• Of greater importance here is eventLeftScope. Put simply, if this is true then we should immediately call PutEvaluation with a value of NOT_APPLICABLE. The most common scenario for eventLeftScope is when a resource is deleted.
• However the real meat on the bones is to be found in the ellipsized invokingEvent. This contains details of the configuration change that triggered this Event. It takes the form of a JSON string representing a ‘Configuration Item’. The docs here explain what that entails: a whole load of information about the underlying resource such as time-of-capture of event, tags, relationships with other resources, and a snapshot of the resource’s current configuration.

With that information suitably digested, lets move on to actually writing a custom check.

The Code

We will write a Lambda handler to process a triggered Config Event, and then deploy it using the SAM framework.

All of the code described below, along with the SAM template and some deployment helpers, is to be found here.

First off, wouldn’t it be good to get our hands on some sample data to gain some degree of confidence in our function before we release it into the wild? Look no further than the AWS CLI. The get-resource-config-history command will, with a little tinkering, furnish us with a Configuration Item that matches what AWS Config will send to our Lambda function.

The following command will, with a sprinkling of jq , give us something to work with (obviously replace the instance-id with one that is real):

$ aws configservice get-resource-config-history \
 --resource-type "AWS::EC2::Instance" \
 --resource-id i-111111111111 \
 --max-items 1 | \
   jq '.configurationItems | .[] | {"configurationItem": .}'

The resulting JSON still requires some doctoring in order to match what we will see in the wild:

• The timestamps configurationItemCaptureTime and resourceCreationTime must be converted to ISO 8601 human-readable time formats.
• The configurationStateId must be converted to a number (just remove the quotes)
• Most importantly, the configuration field must be converted from a JSON string to an actual JSON object.

Failure to carry out any of these tweaks will result in your function imploding the moment it sets foot in the wild.

In the code example, we have saved this patched-up JSON in fixtures/sampleConfigItem.json. We will use this shortly to test our function.

Data Models

The Event that is generated by a triggered Config change is passed into our Lambda handler as JSON. We can take advantage of a few truisms to avoid creating elaborate data structures in which to house it:

• The Event itself is an instantiation of a ConfigEvent. As long as our Lambda function accepts a ConfigEvent as a parameter, it will be automatically un-marshalled.
• Unfortunately we don’t have that luxury with the InvokingEvent, so we are in fact forced to create one elaborate data structure. See here.
• We are told here that the configuration field of the Configuration Item is ‘Information returned through a call to the Describe or List API of the resource’ which means it should nicely un-marshall into an object of the underlying type* (in our case an ec2.Instance). This can be seen in the code example:

// unmarshal the ConfigurationItem
  var instance ec2.Instance
  err = json.Unmarshal(
          invokingEvent.ConfigurationItem.Configuration, 
          &instance)

After this we have an instance object neatly populated with the attributes of the resource that has changed.

(*this should be taken with a pinch of salt. We have found, for example, that the configuration field of EC2 Security Groups does not correlate with what is returned by an API call to DescribeSecurityGroups)

Evaluation Logic

Full code listing for the evaluation of the check can be found here . It’s recommended to have a peruse, but the most interesting part is:

Here we take advantage of the fact that the ConfigurationItem object of the invokingEvent contains a list of the instance’s EC2 Tags, in order to filter for ‘Production’ and ‘database’. If we find a match then we use our instance object to check that EBS optimization has been turned on. The return value is a struct that contains all the information required to complete a PutEvaluation call.

In the same file we provide a CompleteEvaluation function which our Lambda handler will use to actually send the evaluation through to AWS Config.

Remember the sample data we generated above? Here we can see it in action in a unit test that caters for three different outcomes: compliant, non-compliant and non-applicable.

The Handler

Our Lambda handler function is now very simple. It takes the ConfigEvent, sends to our evaluator, and then completes the evaluation:

Deploying

We can easily deploy our custom config rule using SAM. The template looks like this:

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Description: Custom Config Rule

Resources:
  
  DatabaseCheckFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      Runtime: go1.x
      CodeUri: ./main.zip
      Handler: bin/databaseTerminationProtection
      Policies:
      - Statement:
        - Sid: ConfigPutEvaluation
          Effect: Allow
          Action:
          - config:PutEvaluations
          Resource: "*" # No resource level permissions :(

  DatabaseCheckRule:
    Type: AWS::Config::ConfigRule
    DependsOn: DatabaseCheckConfigRole
    Properties:
      ConfigRuleName: "database-ebs-optimization"
      Description: "Checks for production dbs without EBS optimization"
      MaximumExecutionFrequency: One_Hour
      Scope:
        ComplianceResourceTypes:
        - "AWS::EC2::Instance"
      Source:
        Owner: CUSTOM_LAMBDA
        SourceDetails:
        - EventSource: aws.config
          MessageType: ConfigurationItemChangeNotification
        SourceIdentifier:
          !GetAtt
        - DatabaseCheckFunction
        - Arn

  DatabaseCheckConfigRole:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName:
        !GetAtt
      - DatabaseCheckFunction
      - Arn
      Action: lambda:InvokeFunction
      Principal: config.amazonaws.com

The template creates:

• A Lambda function to run our handler, with permissions to make a call to PutEvaluations. We also grant permission for the config.amazonaws.com service to invoke the function.
• A ConfigRule that acts as a trigger to the function. Note that we list “AWS::EC2::Instance” as a ComplianceResourceType. This means that any configuration changes to any EC2 instances will trigger an invocation of the Lambda function.

Exactly how to build and deploy this application is a little out of scope for this article (if you’re interested, see here). However a Makefile is provided, so if you are keen you can just run:

S3_BUCKET=<an-s3-bucket-you-own> make deploy

Once deployed, AWS Config will trigger a run-through of the check for all current EC2 instances. Subsequent checks will only be triggered on configuration changes.

To see the results, navigate to the Config page in the AWS Console; then choose Rules -> database-ebs-optimization. Of course, it’s unlikely that you’ll have any non-compliant resources.

If you were so inclined, you could add an ENVIRONMENT=Production and SERVICE=database tag to an EC2 instance without EBS optimization, then watch the magic happen…(disclaimer: the magic may take a few minutes to happen)

Summary

Hopefully this has given a glimpse of the power of AWS Config Rules in showing that they can immediately notify us of any changes to our estate that have resulted in a breach of Policy. Properly configured they provide a valuable ally in the fight against malicious actors and witless engineers.

Also, we’ve barely scratched the surface here. Happy Rule-Making.