Steps to secure AWS Serverless — Lambda (Part 1)

Serverless is becoming mainstream in business critical applications. At Orchestrated, we make extensive use of serverless in our product suite to cost effectively cater to our enterprise clients especially in the data isolation domain. To learn more about that multi-tenancy journey see http://bit.ly/serverlessScale.

One essential element of productionising an application is security. This is my journey on how to secure the AWS Lambda — serverless compute.

There are four main principles that guide us in our security journey toward achieving Confidentiality, Integrity and Availability — often referred to as the CIA security triad:

1. Apply least privilege
2. Minimise the attack surface
3. Layer the security measures to achieve defence in depth
4. Encrypt everything

Based on these principles and best practices, the following were applied. No doubt, some of these would be applicable to your environment — only principles 1,2 and 3 are covered in this article.

Apply the principle of least privilege to Lambda execution roles

Each Lambda function has an IAM role (an execution role) associated with it. The role provides the Lambda function with access to other AWS resources. Lambda assumes this role when it executes or runs code.

The principle of least privileges can be applied to Lambda functions by allocating only the minimum privileges required by Lambda functions. For example if a Lambda function requires only Write access to DynamoDB tables, the execution role associated with that Lambda function should only be assigned with PutItem permission with specific DynamoDB table names.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:PutItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:ap-southest-2:xxxxxx:table/events"
            ],
            "Effect": "Allow"
        }
    ]
}

Tips: If you are not sure what the full set of minimum privileges should be, start by allocating the absolute minimum privilege to your Lambda function. Then, iteratively, refer to the Lambda logs to identify which permissions are further required to enable the actions to complete. You can then increase permissions one by one. The IAM policy reference is also a very handy guide, as it provides the full action list available for AWS resources. AWS policy generator and IAM policy simulator are good for testing and troubling shooting as well.

Make sure your lambda functions have write access to AWS CloudWatch Logs

It’s best practice to ensure the appropriate access is granted to Lambda logs for debugging and troubleshooting in case of failures in a function. Lambda, by default, automatically integrates with CloudWatch Logs. To provide a Lambda function the appropriate permissions to push logs from your code to CloudWatch Logs, ensure you allow the ability to create a log group, log stream and put events, as shown below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
}
# NB: the resource ARN can be locked down further with region, account or log group name

Avoid sharing the same IAM role between Lambda functions

Ideally, each Lambda function is designed to perform its own unique function, hence each should have it’s own unique set of permissions. However, a problem often arises when multiple functions are initially developed with common permission requirements, and so the same IAM execution role is used.

After further development, more and more permissions are added to this IAM role. Not all of those permissions are required by all of the functions.

The process of re-factoring code to split them out is time-consuming. Best practice is to set up a dedicated IAM role per Lambda function from the start.

Figure 1: Dedicated IAM role per Lambda function with minimum privileges

Expose only API endpoint Lambda functions through the API Gateway

Some applications have multiple Lambda functions running simultaneously to provide a service. From a security perspective, not every single Lambda function needs to be exposed as an API endpoint, especially when they are serving internal functions.

Ensure you expose only Lambda functions that handle the HTTP/S or API requests, to minimise the potential attack surface. The best way to do this is to review API gateway resources and ask yourself “Are these resource methods required? Who is using them?” and then remove unnecessary API exposure.

We’ll cover this in detail in a future article — “How to lock-down the AWS API Gateway”

Figure 2: Expose only required Lambda functions through API gateway

Route Lambda traffic to VPC resources using VPC-enabled Lambda

In some use cases, your Lambda function needs access to AWS resources inside your own VPC such as EC2 instances, RDS instances or Redshift. By default, those resources within a VPC are not accessible by Lambda function unless you have VPC resources with their endpoint configured to be publicly available — route (1)as shown below. In that case, the traffic routes from the Lambda function through the Internet to access your VPC resources.

To minimise the traffic routing over the Internet, we can setup VPC-enabled Lambda — route(2) as shown below, so the traffic is routed inside the AWS network instead of routing over the Internet

Figure 3: VPC-Enabled Lambda (2)

This works as follows — AWS Lambda sets up ENI (Elastic Network Interface) to allow your Lambda function to access VPC resources. The ENI is assigned with a private IP address from the subnet IP range that the Lambda function connected to. All the traffic will then be routed inside the AWS network instead of going through the Internet.

Lambda scales the number of ENIs, depending upon traffic. Therefore Lambda needs the ability to create and delete ENIs.

Below is an example of an IAM policy that provide Lambda function the ability to create and delete ENIs when connecting to the VPC

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        - 'ec2:CreateNetworkInterface'
        - 'ec2:DescribeNetworkInterfaces'
        - 'ec2:DeleteNetworkInterface'
      ],
      "Resource": "arn:aws:ec2:*:*:*"
    }
}
#NB: Resource arn can be restricted to region, account and network interface type

To set up VPC-enabled Lambda, two parameters need to be defined, which are the security group ID and subnet ID.

The following is an example using Serverless Framework to configure VPC-enabled Lambda.

functions: 
    lambdaFunc1:
        role: lambdaFunc1Role
        handler: handler.lambdaFunc1handle
        vpc:
            securityGroupIds: 
                - Fn::GetAtt: [ lambdaFunc1SG, GroupId ]
            subnetIds: 
                - ${self:subnetID}

Apply the principle of least privilege to the Lambda function’s security group

The security group of the Lambda function controls the inbound and outbound traffic from the ENI attaching to the Lambda functions.

For example, if the Lambda function connects to EC2 in private subnet 10.0.78.64 on port 6231, the outbound rule of the security group should allow only outgoing port 6231 as per the principle of least privilege. An example of outbound rule of Lambda function’s security group as shown below.

Figure 4: Outbound rules for VPC-enabled Lambda

What if my VPC-Enabled Lambda function needs access to Internet

By default, after you enable VPC for Lambda, access to the Internet is removed. If your Lambda function needs access to Internet, then two elements need to be addressed:

1. Set up the security group to allow the outgoing traffic to access the Internet as shown in the table below(row 2).

2 . Configure your subnet to enable routing to the Internet

Figure 5: Add outbound rules to allow Lambda an access to the Internet

The subnet that Lambda connects to needs to be configured for outbound Internet access. As per the diagram below, if Lambda connects to a private subnet, then we need to configure the route table of that private subnet to route the traffic to a NAT gateway in the public subnet. The NAT gateway routes the traffic to the Internet via a router and Internet Gateway.

Figure 6: Outbound Internet access for VPC-enabled Lambda

An example of a private subnet’s route table that allows outgoing connection to the Internet is shown below.

Figure 7: Route table of private subnet for outbound Internet access

Routing Lambda traffic to AWS resources using a VPC-enabled Lambda function and a VPC endpoint

If a Lambda function requires access to Amazon S3 and DynamoDB tables, then by default, the traffic routes from the Lambda function over the Internet to S3 and DynamoDB. To minimise traffic over the Internet, the following three steps below are required, so the traffic can route inside the AWS network from end-to-end (from 1–2–3 for Function A and 4–5–6 for Function B as shown below)

Figure 8: VPC-Enabled Lambda with VPC endpoints

Step 1: Configure VPC-enabled Lambda functions.

VPC-enabled Lambda functions route their traffic through the AWS network instead of the Internet. The policies attached to Lambda roles control access to AWS resources. As shown below, the policy only allows Write access to the DynamoDB table via the DynamoDB VPC endpoint. This means that any Lambda functions which assume this role can access DynamoDB tables via this VPC endpoint. Any connection via Internet or other VPC endpoints will be denied.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:PutItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:ap-southeast2:xxxxxx:table/events"
            ],
            "Effect": "Allow"
            "Condition":
                  StringEquals:
                    'aws:sourceVpce': "vpce-xxxxx"
        }
    ]
}

Step 2: Configure routing from a specific subnet to a VPC endpoint.

Routing is another security mechanism; no route means no traffic. We should only enable routing for specific subnets to specific endpoints. For example, we permit the subnet with a route to the DynamoDB VPC endpoint (as shown in row 2 of routing table below). Hence only Lambda functions connecting to this subnet can then route to DynamoDB.

Figure 9: Routing table with a route to DynamoDB VPC endpoint

Step 3: Update the policy of the VPC endpoint.

The VPC endpoint policy controls an access to services that connecting to the endpoint. For example, the policy below is configured with least privilege principle to limit the Write traffic to only the Events table.

In the situation where both Read and Write access is inappropriately granted for the Lambda function’s role, when a Lambda function tries to read data from the Events DynamoDB table, the endpoint policy will block the Read request.

As per our third principle, if the first layer of defence fails, we have subsequent layers to protect us.

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "dynamodb:PutItem",
            ],
            "Resource": "arn:aws:dynamodb:ap-southeast-2:xxxxx:table/events"
        }
    ]
}

Checkpoint

Action 1: Review all IAM policies attached to Lambda roles and ensure they have only minimal privilege policies

Action 2: Ensure your Lambda functions can write to CloudWatch Logs for troubleshooting

Action 3: Avoid sharing the same IAM role between Lambda functions

Action 4: Review API gateway resources associated with Lambda functions and ensure there are no extraneous API resources.

Action 5: Review how your Lambda functions access your VPC resources and consider using VPC-enabled Lambda functions

Action 6: Review your Lambda functions’ security groups, ensuring they have minimal privilege policies

Action 7: Review how your Lambda functions connect to DynamoDB and S3. Consider using VPC-enabled Lambda functions and VPC endpoints

Action 8: Avoid associating all subnets with VPC endpoints, associate only subnets that need access to required endpoint resources

Action 9: Lock down VPC endpoint policies to only permitted actions or resources

In Part 2, I will outline further techniques to secure AWS serverless Lambda functions.

We’re always interested in learning — what have you adopted to secure your serverless environment? Please get in touch or tell us in the comments!