Using AWS X-Ray to achieve Least Privilege Permissions — Part 2

How to use AWS X-Ray to achieve Least Privilege IAM Permissions during integration testing and continuous monitoring of staging/production environments.

This is part two on how to use the aws-least-privilege open source tool. Project is available on GitHub at: https://github.com/functionalone/aws-least-privilege. In part one, I gave a walk-through of how the open source tool can be used to generate a Least Privilege AWS IAM policy based upon collected X-Ray traces. You can read part one here:

Using AWS X-Ray to achieve Least Privilege Permissions

In this walk through I would like to show how this tool can be used during CI pipelines and also for continuous monitoring of staging/production environments. The example is focused on Lambda functions but same techniques can easily be applied to other applications that utilize AWS Roles (EC2 or ECS).

Compare Mode

The aws-least-privilege tool provides a compare mode command line option. With compare mode, the tool will automatically check what role is currently attached to a Lambda Function. It will then compare the role’s permissions against the permissions discovered through the X-Ray scan. If any of the roles contain excessive permissions as compared to the actual permissions discovered in the X-Ray scan, an excessive permissions report in json format will be generated. Compare mode is key for automating the process of discovering excessive permissions.

To use compare mode you first need to make sure the user/role running the cli tool has proper permissions. Beyond the basic AWSXrayReadOnlyAccess policy the tool will also need permissions to read the Lambda configuration to resolve what role the Lambda function is running under and additionally read the role and policy details. To enable these permissions, it is easiest to attach the following inline policy to the user/role that runs the cli tool:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRolePolicy",
                "lambda:GetFunctionConfiguration"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Once the permissions are configured, we can now run the tool with the “-c” command line option. For example to scan the last 12 hours of X-Ray traces and receive a comparison report we can run the tool with the following command line:

xray-privilege-scan -c -r 720

Example Application

For the walk through we’ll use the example application from Serverless Stack (same one we used in Part 1). We’ll use the application already setup with IAM roles per function, but with a mis-configuration on the delete function. The application is configured with per-function IAM roles (using the super easy serverless-iam-roles-per-function plugin). But, the delete function was configured with an unused permission: “dynamodb:GetItem”. You can view the full source code of the mis-configured project here. I’ve also added a simple script to invoke the functions and simulate a very simple integration test. Script is available here. We can now invoke the functions and simulate an integration test. After the functions are invoked, X-Ray traces are created and we can scan the traces to receive a comparison report and least privilege policies per function. After invoking the tool we will see an output of the form:

We can see that an excessive_permissions.json file was generated. This file contains a json report with the permissions that were not used according to the X-Ray traces scanned. The file contains the unused permission assigned to the delete function including full information about the role and function arn:

If multiple functions had been detected with excessive permissions they would have been all included in this json file.

This form of comparison allows us to integrate the tool into a CI flow, assuming there is good test coverage for the Lambda functions during integration testing. The CI can run the integration tests and then run the tool to scan the X-Ray traces. If an excessive permissions report is generated the CI can act upon it (break the build/notify the developer).

The compare mode can also be used for continues monitoring of a staging or production environment. The tool can be used to run on a schedule. A significant time frame should be chosen in which the functions are executed with enough variance to provide coverage regarding what resources the functions access (for example once a day). Once the tool is executed in compare mode, if an excessive permissions report is generated it can be sent out for review and analysis.

While performing continues monitoring, you may wish to scan only specific X-Ray traces. For this scenario, we added an option to specify a filter expression, which can be used to limit the X-Ray traces which are scanned. The filter expression is specified with the cli option of -f. For example, to scan only for the Lambda function which is named notes-app-api-test-delete, you use the following filter: service(id(name: “notes-app-api-test-delete”)). Full documentation about the X-Ray filter syntax is available in the AWS X-Ray Developer Guide.

Conclusion

Using “compare mode” allows us to streamline the process of collecting run-time resource usage information from X-Ray and reaching a Least Privilege IAM Policy for a given application. We can use this method in either as part of integration testing or for continues monitoring of staging/production environments.

We love to hear suggestions and feedback. Please feel free to open an issue on the Github project page with suggestions, bugs, and questions.