Automatic Secret Rotation of AWS-RDS using Lambda

AWS Stack to achieve secure password access and rotation

Through this article, we will be setting up password management for RDS using Secrets Manager with automatic rotation enabled. In this walkthrough we will be setting up our RDS in a private VPC.

Pre — Checklist

1. AWS Account
2. Terraform
3. Python 3.7
4. Hot/Cold beverage to keep you company through the upcoming arduous hours

Today, we will be covering the following aspects —

1. Setting up RDS
2. Setting up Secrets Manager Resource & Rotator Lambda
3. Configuring roles and permissions for AWS Resources
4. Lambda Code for Password Rotation
5. Lambda Code to retrieve secret from Secrets Manager to connect to RDS

Step — 1 Setting up RDS

Let’s start with the terraform code to set up a basic RDS in AWS. Since, we are setting up our RDS in a private VPC, you will notice that the vpc_security_group_id is configured here.

1. Setting up the private subnets: Since, the RDS will be set up in a private VPC, we need to make sure that the Instance has the required subnets associated with the VPC . This will be achieved by creating a subnet resource first which will then be referred in the RDS resource creation step

resource "aws_db_subnet_group" "subnet_group" {
  name       = "main"
  subnet_ids = ["Subnet_id_1", "Subnet_id_2"]
}

2. Setting up the RDS DB Instance: Use the security group ids based on the Security Groups associated with your private VPC . Additionally, ensure to point the db_subnet_group_name attribute to the above created subnet_group resource

resource "aws_db_instance" "test_db" {
  db_subnet_group_name = aws_db_subnet_group.subnet_group.name   
  allocated_storage    = 64
  engine               = "postgres"
  identifier.          = "rds-instance"
  instance_class       = "db.t2.micro"
  name                 = "postgres"
  username             = "admin_user"
  password             = "foobarbaz"
  iam_database_authentication_enabled = true
  vpc_security_group_ids = ["Sgs_1", "Sgs_2"....]
}

Step — 2 Setting up Secrets Manager & Rotator Lambda

We will now set up the Secret Manager resource. Since, the RDS sits in a private VPC , we will first create a custom lambda configured with the required network details to facilitate the communication between Secrets Manager and the RDS.

1. Custom Lambda in private network—

• For a lambda in private VPC to communicate with the Secrets Manager, which by default resides on public Internet, ensure to have a NAT gateway security group in your configuration
• For more information click the below link —

Configuring Your Network to Support Rotating Secrets

• Note: the role here points to the IAM policies we will be creating in the third step of this article

data "archive_file" "secret_rotator_zip" {
  type        = "zip"
  source_file = "${path.module}/init.tpl"
  output_path = "${path.module}/files/init.zip"
}
resource "aws_lambda_function" "secret_rotator_lambda" {
  filename      = "lambda_function_payload.zip"
  function_name = "lambda_function_name"
  role          = [aws_iam_policy.lambda_invoke_policy.arn,                   aws_iam_policy.secrets_admin_policy.arn,                   aws_iam_policy.vpc_policy.arn]
  handler       = "lambda_handler.rds_pwd_rotator"
  filename      = data.archive_file.secret_rotator_zip.output_path
  source_code_hash =   data.archive_file.secret_rotator_zip.output_base64sha256
  runtime       = "python3.7"
  vpc_config {
      security_group_ids = ["Sgs_1", "Sgs_2"....],
      subnet_ids         = ["Subnet_id_1", "Subnet_id_2"]
}

2. Lambda Invoke Permissions for Secrets Manager

Provides the permission to enable invocations on the lambda

resource "aws_lambda_permission" "allow_secret_manager_call_Lambda" {
    function_name = "${aws_lambda_function.secret_rotator_lambda.function_name}"
    statement_id = "AllowExecutionSecretManager"
    action = "lambda:InvokeFunction"
    principal = "secretsmanager.amazonaws.com"
}

3. Create the Secrets Manager Resource

The secret manager resource configured with the lambda responsible for rotating the password

resource "aws_secretsmanager_secret" "secret" {
  description         = "Secret Manager for RDS"
  name                = "secret-mgr-rds"
  rotation_lambda_arn = aws_lambda_function.secret_rotator_lambda.function_name
  rotation_rules {
    automatically_after_days = 7
  }
}

4. Create the Secret Manager Version Management Resource

This resource sets the secret string in the above secret manager. It additionally, maintains the versioning of the password to identify status of previous and current password.

resource "aws_secretsmanager_secret_version" "secret" {
  secret_id     = "${aws_secretsmanager_secret.secret.id}"
  secret_string = <<EOF
{
  "password": "initial_password"
 }
EOF
}

Step — 3 Role and Permission Management for AWS Resources

1. Roles for the Lambda rotating the password: All the below mentioned policies needs to be created via aws_iam_policy resource and then attached against the role attribute of the concerned lambda.

a. Lambda Invocation —

This sets the permission to allow the rotator lambda to be invoked by the Secrets Manager setup

data "aws_iam_policy_document" "lambda_invoke_policy" {
  statement {
    sid = "Invoke Lambda"

    actions = [
      "lambda:GetFunction",
      "lambda:InvokeAsync",
      "lambda:InvokeFunction"]

    resources = [
      "arn:aws:lambda:::*",
    ]
  }
resource "aws_iam_policy" "lambda_invoke_policy" {
  name   = "Lambda-invoke-policy"
  policy = "${data.aws_iam_policy_document.lambda_invoke_policy.json}"
}

b. Secrets Manager Admin Privilege —

Admin privileges are given to enable the rotator lambda to have full control in changing the secret set in Secrets Manager

data "aws_iam_policy_document" "secrets_admin_policy" {
  statement {
    sid = "Secrets Manager Admin"

    actions = ["secretsmanager:*"]

    resources = [*]
  }
resource "aws_iam_policy" "secrets_admin_policy" {
  name   = "secrets-admin-policy"
  policy = "${data.aws_iam_policy_document.secrets_admin_policy.json}"
}

c. Network Related ENI permission associated with VPC —

This enables Lambda to set up the required configuration when you specify running your database or service in a VPC

data "aws_iam_policy_document" "vpc_policy" {
  statement {
    sid = "VPC ENI policies"

    actions = [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterfaces",
"ec2:DeleteNetworkInterfaces"]

    resources = [*]
  }
resource "aws_iam_policy" "vpc_policy" {
  name   = "vpc-secret-mgr-policy"
  policy = "${data.aws_iam_policy_document.vpc_policy.json}"
}

2. Roles for lambda accessing secret Manager for retrieving the credentials —

For a lambda that wants to merely read the encrypted secret string present in the secrets manager, we can provide constrained policy as below to achieve that

data "aws_iam_policy_document" "secrets_mgr_read_policy" {
  statement {
    sid = "Secrets Manager Read Policy"

    actions = ["secretsmanager:getSecretValue"]

    resources = [*]
  }
resource "aws_iam_policy" "secrets_mgr_read_policy" {
  name   = "secret-mgr-read-policy"
  policy = "${data.aws_iam_policy_document.secrets_mgr_read_policy.json}"
}

Step — 4 Setting up Secrets Manager Rotation Code for Lambda in private VPC

For our purpose we will be using postgres single rotator python code which is made available in the below link by AWS

aws-samples/aws-secrets-manager-rotation-lambdas

Step — 5 Creating a lambda to connecting to the RDS by retrieving information from the Secrets Manager

Sample code for connecting to the Secrets Manager to create connection string will be as given below

import boto3
import json
from botocore.exceptions import ClientError


def get_secret():
    secret_name = "MySecretName"
    region_name = "us-west-2"

    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name,
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except ClientError as e:
        if e.response['Error']['Code'] == 'ResourceNotFoundException':
            print("The requested secret " + secret_name + " was not found")
        elif e.response['Error']['Code'] == 'InvalidRequestException':
            print("The request was invalid due to:", e)
        elif e.response['Error']['Code'] == 'InvalidParameterException':
            print("The request had invalid params:", e)
    else:
        # Secrets Manager decrypts the secret value using the associated KMS CMK
        # Depending on whether the secret was a string or binary, only one of these fields will be populated
        if 'SecretString' in get_secret_value_response:
            text_secret_data = get_secret_value_response['SecretString']
        else:
            binary_secret_data = get_secret_value_response['SecretBinary']
SecretString = json.loads(SecretString)
password = SecretString["password"]
# Your code goes here to make connection string using the above fetched password

Disclaimer — Currently, in the latest version of terraform there is a bug due to which the password does not rotate immediately when the resources are created. Further information can be found in the below issue. The issue is expected to be resolved in the upcoming releases.

New resource: aws_secretsmanager_secret_rotation by kelvin-acosta · Pull Request #9487 ·…

Suggested workaround is to manually rotate the password the first time via following two options post which the other rotations will happen automatically—

1.Trigger via AWS CLI Command —

aws secretsmanager rotate-secret --secret-id <name of the secrets manager resource>

2. Trigger via AWS Console —

a. Go to Secrets Manager on AWS Console

b. Search for your secrets manager resource by name and click on it

c. Click on “Rotate Secret Immediately” under “Rotation Configuration”

Github Link to the repo —

ayshamdmgit/AWS-SECRETS-MANAGER-RDS

Note — The network part of the code including security groups and VPC configs consists of dummy data

Secrets Manager Guide — For more details refer to the official guide linked below

Follow Up Article Alert — There will be a follow up article where we will walk through how to connect to the RDS using IAM authentication. This will improve the security by enabling password-less connections.