Elevating Security Standards: Automating IAM User Access Key Rotation

AWS security best practice by automating IAM user access key rotation with AWS Lambda, Organizations, and Secrets Manager

Published in

DevOpsars

6 min readMay 7, 2024

Disclaimer

The information provided in this article is based on my experience implementing the solution, and it is based on the documentation provided by AWS here: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-rotate-iam-user-access-keys-at-scale-with-aws-organizations-and-aws-secrets-manager.html

The user must thoroughly test and understand the solution’s impact before implementing it in production. The author and the original authors of the solution are not responsible for any loss of access, broken code, or system instability that may result from using the solution. The user is responsible for ensuring that the solution is implemented correctly and that all necessary backups and safeguards are in place.

Introduction:

Automating the rotation of IAM user access keys is a crucial step in maintaining the security of your AWS environment. With the increasing number of users and resources in an AWS environment, it becomes increasingly difficult to rotate access keys, especially at scale manually.

This article provides a guide to automatically rotating IAM user access keys in an AWS environment using AWS Organizations, AWS Secrets Manager, AWS CloudFormation templates, and Lambda functions. It provides a solution to prevent a compromised set of IAM access keys from accessing components in your AWS account by generating new access keys when existing ones are 90 days old, deactivating previous access keys at 100 days old, and deleting them at 110 days old.

The solution can be deployed in single or multiple accounts and generates centralized email notifications.

Problem statement

Manually rotating IAM user access keys can be a tedious and error-prone task, especially when dealing with many users and resources. In addition, it can be difficult to track the age of access keys and ensure that they are rotated at regular intervals. This can lead to security vulnerabilities, as access keys that are not rotated in a timely manner can be compromised.

Solution

AWS Organizations and Secrets Manager provide a solution to automatically rotate IAM user access keys at scale with the help of Lambda functions. We can centrally manage access keys across multiple AWS accounts and regions by using AWS Organizations. At the same time, AWS Secrets Manager allows you to store access keys securely. In contrast, Lambda functions generate inventory for the access keys, automatically invoke key rotation, and send notifications to the admin. Together, these services provide a secure and efficient way to automate the rotation of IAM user access keys.

Prior to proceeding, please ensure that the following prerequisites are met.

An active AWS account is required.
AWS Organizations must be configured and set up (see tutorial).
Permission to query AWS Organizations from your management account is necessary. Refer to the AWS Organizations documentation for more information on AWS Organizations and service-linked roles.
An IAM principal with the necessary permissions to launch the AWS CloudFormation template and associated resources must be in place. For further details, refer to the section on Granting self-managed permissions in the AWS CloudFormation documentation.
An existing Amazon Simple Storage Service (Amazon S3) bucket must be available to deploy the resources.
Amazon Simple Email Service (Amazon SES) must be moved out of the sandbox.

Implementation

Implementing the solution requires the execution of the following steps in a precise and orderly manner to ensure optimal results.:

Clone the code repo to the local machine https://github.com/aws-samples/aws-iam-access-key-auto-rotation

IMPORTANT: There is a minor bug in the name of the “template” folder under this repo; it has to be in “Template” in upper case T this requires modifying the name before the next step

2. Create an S3 bucket with any name we want, and upload the necessary files to the bucket (bucket structure: asa/asa-iam-rotation). The folders to upload are: CloudFormation, Lambda, Test Units, Template

5. Launch and configure the CloudFormation Stacks for the primary and secondary account(s). This steps

5.1 Deploy ASA-iam-key-auto-rotation-and-notifier-solution.yaml on the main account

Here is how the parameters look like after completing the configuration

Explaining some of the important parameters

CloudFormation S3 Bucket Name (S3BucketName) – The name of the deployment S3 bucket that contains your Lambda code.
Dry Run Flag (Audit Mode) (DryRunFlag) – True to turn on audit mode (default) False to turn on enforcement mode. IAM access keys aren’t modified in audit mode, but an email is sent to notify users.
AWS Organization ID (AWSOrgID) – The unique ID of your organization, which begins with o- and is followed by 10-32 lowercase letters or digits. You can find it by visiting the Organization page on the AWS console on the main account.
Account to List Organization Accounts — Enter the main account ID, which will be used to list Organization accounts.
Admin Email Address (AdminEmailAddress) – A valid email address to send notifications to.
Assumed IAM Role Name (IAMRoleName) – Keep the default.
VPC Id for Lambda functions (VpcId), VPC CIDR for Security Group Rule(VpcCidr), and Subnet Id for Lambda functions (SubnetId) – I’m not going to use this feature as it is optional, but we need to make sure to set RunLambdaInVpc to False.
SMTP User SSM Parameter Name (SMTPUserParamName) and SMTP Password SSM Parameter Name (SMTPPasswordParamName) – Used with the VPC option.
Secrets Store flag for central account –False to store the new access keys on each member account (option 1 below). True to store the new access keys on the management account (option 2 below)

The solution supports two scenarios for storing the credentials: in a member account and the management account.

Option 1: Store the credentials in a member account

Option 2: Store the credentials in the management account

b. Next, we will deploy ASA-iam-key-auto-rotation-iam-assumed-roles.yaml to the main account as stack and on the other accounts as StackSet. Keep the default values as they are. The only thing to modify is the main account ID, which is our management account, as described before.

Deploy the ASA-iam-key-auto-rotation-iam-assumed-roles.yaml template for each account where you want to rotate keys. If you have multiple accounts, you can deploy the main CloudFormation template in your management account as a stack and deploy the ASA-iam-key-auto-rotation-iam-assumed-roles.yaml template with CloudFormation stack sets to all required accounts. Make sure to deploy it on the main account as a stack, even if you deployed it as Stackset.

c. Then, the ASA-iam-key-auto-rotation-list-accounts-role.yaml template will be deployed in the deployment account. Keep the default values as they are. The only thing to modify is the main account ID, which is our management account, as described before.

6. For IAM users to be able to retrieve their access keys from the secret manager, we need to add a new IAM Policy and attach it to the intended users/ groups. This step wasn’t mentioned in the official documentation and is based on my experiment for this solution.

The IAM Policy is

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GetResourcePolicy",
            "Effect": "Allow",
            "Action": "secretsmanager:GetResourcePolicy",
            "Resource": "arn:aws:secretsmanager:*:ACCOUNT_NUMBER:secret:*"
        },
        {
            "Sid": "ListSecrets",
            "Effect": "Allow",
            "Action": "secretsmanager:ListSecrets",
            "Resource": "*"
        }
    ]
}

7. Now, we will do a dry run to ensure the process works as expected. (unit test)

To run the test, go to AWS Lambda service, choose the ASA-IAM-Access-Key-Rotation-Function function, go ahead to the Test tab, scroll down to the Event Json text box, and paste the JSON object below as described.

{
  "account": "AWS ACCOUNT ID",
  "name": "ACCOUNT NAME",
  "email": "EMAIL ADDRESS"
}

account: is the account number that we need to examine. This value is mandatory and must be one of the accounts under the organization or the main account.
name: the account name (optional)
email: this is optional as well

See the screenshots below for clarification: