Threat Modeling — Lambda

Embedding security into the lambda service offering

In the last blog, Threat Modelling — EKS, we saw how to evaluate the security posture of EKS, in this blog, we will do a similar exercise for AWS Lambda.

Shared responsibility Model

Customer

Function code, libraries, extension, layers
Data in cloud
Data in Transit
Resource Configuration
IAM
Other Service configuration

AWS:

Compute
Networking
Execution environment
Runtime language
Server software Hardware AZ, Region

The below diagram depicts the various actors, entry, and exit points of the lambda service.

Dataflow of lambda (security view)

Resources

• Lambda Function
• Lambda Function URL
• Lambda Layer
• Lambda EventSourceMapping Object
• Lambda DLQ & EventDestinations
• Lambda Version & Aliases
• Lambda Resource Policy
• Lambda EFS mounts
• Lambda Service Endpoint
• ECR for Docker images

Threats

• Improper Access Control
• Privilege Escalation
• Insecure resource configuration
• Inadequate Function monitoring & logging
• Improper exception handling & verbose error
• Sensitive data exposure
• Insecure 3rd Party Dependency
• Event payload manipulation
• Insecure secret storage
• Resource exhaustion
• Execution flow manipulation

After looking at the threat vectors, we have the following pointers to ensure secure lambda consumption:

Set access control standards and limit access to Lambda APIs and deployment process — We should avoid creating lambda function from the console. It should be mapped to CI/CD process using IaC. This will allow us to control admin APIs for lambda to a few roles. We also need to restrict the roles that can be passed to lambda functions. The APIs in question are the following:

lambda:AddPermission lambda:CreateAlias lambda:CreateEventSourceMapping lambda:CreateFunction lambda:CreateFunctionUrlConfig lambda:DeleteEventSourceMapping lambda:DeleteFunction lambda:DeleteFunctionEventInvokeConfig lambda:DeleteLayerVersion lambda:InvokeFunction lambda:InvokeFunctionUrl lambda:PublishLayerVersion lambda:PublishVersion lambda:PutFunctionEventInvokeConfig lambda:RemovePermisison lambda:UpdateAlias lambda:EventSourceMapping lambda:UpdateFunctionCode lambda:UpdateFunctionConfiguration lambda:UpdateFunctionEventInvokeConfig lambda:UpdateFunctionUrlConfig

Avoid monolambdas — By keeping the functions granular, we can limit their privileges so that surface of attack is reduced. One Lambda function should serve one purpose and it should be mapped to a role that is exclusive to this lambda.

Grant least privilege — We need to grant only the required access to the function role. For example, if the function only needs read-write access to an S3 bucket path, we should only grant that. We should be specific with API actions and resources.

Reserved concurrency — For critical functions ensure the concurrency is reserved. This helps us in isolating the function from other functions. Also, this sets the upper limit, so that in the case of DoS attacks function is not running away with the account concurrent invocation limit. It also improves the cold starts. Provisioned Concurrency of 5 will keep 5 execution environments prepared ahead of function invocation.

Implement Circuit breaker — For event source mapping objects we should always implement a circuit breaker pattern. Based on custom CloudWatch metrics we should enable or disable the event source mapping. This pattern prevents DDoS using source flooding.

RDS Proxy with lambda — Using an RDS proxy to connect the database allows us to pool connections and we don't need to pass around DB credentials. rds-dc:connect

Event Data Validation — We should leverage the event validation framework in lambda code to ensure that the lambda is getting the expected event. We can leverage pydanticfor python to do event validation.

pip install aws-lambda-powertools[pydantic]

Remove PII from logs — Since log/print statements will put the data in the CloudWatch we should avoid doing it for data. Only execution flow info/error should be logged.

Implement Dependency scanning in the build process — We don’t want insecure code in production. So we want dependency scan as part of our build process e.g. Lambda layers and lambda code. Lambda layers need special attention as it also used for extensions. Packages like trivy & safety can be used to do the scanning.

pip install safety
safety check -r requirements.txt --full-report # manual
safety check -r requirements --bare # get vulnerable package & fail

Image scanning — Similar to dependency scanning we should do continuous scanning of ECR images used in the lambda function. twistlock can be added as a scanning tool during the image build process.

Restrict access to ECR — Since lambda uses the same role to provision and execute the function, we should restrict access to the ECR (for docker function) only to specific image repo.

Restrict access to EFS mounts

No Credentials in Lambda function code or configuration — We should always get the credentials (third-party API creds) from the system manager or secret manager. No hardcoding in the function code or env.

Delete unused functions— It's always good to clean up unused lambda functions. This reduces attack vectors as older lambdas may have outdated and unpatched runtimes.

Versioning & Aliasing — We should avoid using $LATEST. This is the default and easy to guess. The better option is to use versioning and add an alias to it. Grant execute permission on aliases. This helps in reducing the hit & trial exploits.

{
  "Sid": "Invoke",
  "Effect": "Allow",
  "Action": [
    "lambda:InvokeFunction"
  ],
  "Resource": "arn:aws:lambda:us-east-1:123456789012:function:our-function:cool-alias"
}

Secure Dead letter Queue — Since DLQ is used to pass failed events, we need to secure DLQs as they might have sensitive events.

Remove deprecated runtimes — Since AWS does not apply security patches to deprecated runtimes e.g. python 2.* python 3.4|5|6, we should not use those runtimes.

Encrypt environment variables — If we are making our lambda parametrized using env we should encrypt those variables using KMS so that those are not visible in the console as well as when the lambda function is described by unauthorized roles. Also, store sensitive information in SSM/Secret Manager and ask for it on demand, do not log or print these in function code.

Lambda behind API Gateway (Implement Auth) — Direct invocation of lambda functions should be avoided. We should put them behind API Gateway and implement auth at API Gateway. For lambda behind ALB, we should have authentication and authorization logic in lambda code. Also for ALB, we should implement IP & protocol boundaries. For lambda with function URL ensure AUTH is enabled. Also, we should restrict the lambda:InvokeFunctionUrlAPI to as few roles as possible.

Lockdown VPC endpoint policies to only permitted actions or resources — It is a good point to add endpoint policies for AWS services to be as restrictive as possible. Also, add a condition for invocation using vpce for lambdas which are in VPC

{
  "Action": ["lambda:InvokeFunction"],
  "Resource": [
    "arn:aws:lambda:us-east-1:123456789012:function:our-function:cool-alias"
  ],
  "Effect": "Allow"
  "Condition":
    StringEquals:
     'aws:sourceVpce': "vpce-xxxxx"
}

Short function timeouts — Smaller timeout in functions reduces the risk of longer runtime of malicious code. But make sure your timeout accounts for certain delayed execution. P95 duration + 10%.

Clean /tmp — Between back-to-back sequential invocations /tmp folder is reused, so it is imperative to clean up the files written on /tmp during function execution. As it might have sensitive information. Use runtime temporary files with context handlers, like tempfile.

Logging & Monitoring — We should ensure that lambda logging is enabled and for production workload, we have enhanced monitoring. We should create alerts on Lambda failure, timeout, & throttling.

Active Tracing — For lambdas used with other lambdas or serverless components we should enable Trace with X-ray. This allows us to see a service map as well as track execution flow end to end.

Detective Rule

We can create Config rules to be proactive to security issues as soon as possible. The following config rule can be used:

Check Lambda Functions with Wildcard IAM Permissions
Check Lambda function without Dead Letter Queue
Check Function URL without AUTH
Check Lambda function without trigger
Check Lambda function with multiple trigger
Check Unknown Cross-Account Access
Check Lambda function with deprecated Runtime
Check for Single IAM Role being used by more than one Lambda Function
Check Lambda Functions that are created using console(CloudTrail rule)

Best Practices

Graviton runtime to save cost
Deploy common code to lambda layer
Smaller Docker image
Separate lambda handler from service logic
Enable DLQ for lambdas
Prudent Retention setting for lambda logs
Utilize Lambda Powertools

Happy Security !!

Resources:

Homepage

pydantic