Automatic password rotation 🔐

How we no longer have to worry about security

“PASSWORDS ARE LIKE UNDERWEAR: YOU DON’T LET PEOPLE SEE IT, YOU SHOULD CHANGE IT VERY OFTEN, AND YOU SHOULDN’T SHARE IT WITH STRANGERS“ (Chris Pirillo).

“What if you could change underwear automatically?” is what went through my head when someone criticises the red exclamation marks in AWS Identity and Access Management (IAM).

snapshot of the access key age before and after the automatic password rotation

Setting up auto-rotation of credentials in K8s on AWS

Our services run in containers in Kubernetes (K8s). We use AWS Elastic Kubernetes Service (EKS) as the service to manage this. Our universe of services and tools also includes AWS IAM to manage our technical users and AWS Secrets Manager to manage all the credentials for our services. To get the credentials from AWS Secrets Manager to K8s we use a tool called Kubernetes External Secrets.

external-secrets/kubernetes-external-secrets

The technical user has security credentials (access key ID and secret access key). With these credentials every technical user has corresponding permissions. For example: We use one technical user to read and write objects in one specific AWS S3 bucket. The given permissions are only for this specific S3 bucket and the user can not do anything else. If we create the user with Terraform, we store the credentials in a new secret inside the Secrets Manager:

resource "aws_iam_user" "default" {
  name = "${var.environment}-${var.team_name}-${var.bucket_name}-s3-user"
}
resource "aws_iam_access_key" "default" {
  user = aws_iam_user.default.name
}
resource "aws_secretsmanager_secret" "secret" {
  name = "${var.environment}/${var.bucket_name}-s3"
  description = "This is a terraform generate secret for the s3 bucket ${var.bucket_name}. ATTENTION: DO NOT MODIFIED BY HAND."
}

resource "aws_secretsmanager_secret_version" "secret_value" {
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = jsonencode(map("access-id", aws_iam_access_key.default.id, "secret-key", aws_iam_access_key.default.secret))
}

Our services are defined by Helm Charts and deployed in the K8s cluster by Flux. An external secret yaml is defined in the Helm Chart, so that the Kubernetes External Secrets service knows which secret it has to pack where:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: {{ .Chart.Name }}-secrets
  namespace: {{ .Release.Namespace }}
spec:
  backendType: secretsManager
  dataFrom:
    - {{ .Values.environment }}/binary-leancms-s3

Our service found the K8s Secret with the credentials and thereby gets access to the AWS S3 bucket. All of our access to AWS resources work according to this principle:

An overview of the various services and how the credentials get into the K8s cluster

Activating auto rotation of the security credentials

The AWS Secrets Manager can configure to automatically rotate the secrets. To do this, it calls a specified lambda for a defined cycle. In our case every 30 days. The code for the lambda in this case, is provided by us. The lambda has permission for the technical user and the corresponding secret inside the Secrets Manager.

resource "aws_secretsmanager_secret_rotation" "default" {
  secret_id           = aws_secretsmanager_secret.secret.id
  rotation_lambda_arn = module.iam-secret-rotate-lambda.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

For the lambda code, i wrote a little Terraform module, which in turn uses the official Terraform AWS lambda module.

data "archive_file" "default" {

  type        = "zip"
  output_path = "${path.module}/src/package.zip"

  source {
    content  = templatefile("${path.module}/src/index.tpl", { iam_user_name = var.iam_user_name, secret_name = var.secret_name, sns_topic_arn = var.sns_topic_arn})
    filename = "index.py"
  }
}

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"
  version = "v1.24.0"

  create = var.create
  function_name = "${var.environment}-${var.team_name}-${var.service_name}-iam-secret-rotate-lambda"
  description = "A function to rotate IAM Access Keys and Secrets Manager Secrets"
  handler = "index.lambda_handler"
  runtime = "python3.8"
  create_role = true
  attach_policy_json = true
  policy_json = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecretManager",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:DeleteSecret",
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:UpdateSecret",
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:RestoreSecret",
                "secretsmanager:UpdateSecretVersionStage",
                "secretsmanager:RotateSecret"
            ],
            "Resource": [
                "${var.secret_arn}"
            ]
        },
        {
            "Sid": "IAM",
            "Effect": "Allow",
            "Action": [
                "iam:DeleteAccessKey",
                "iam:UpdateUser",
                "iam:GetAccessKeyLastUsed",
                "iam:UpdateAccessKey",
                "iam:CreateAccessKey",
                "iam:GetUser",
                "iam:ChangePassword",
                "iam:ListAccessKeys"
            ],
            "Resource": [
                "arn:aws:iam::${var.account_id}:user/${var.iam_user_name}"
            ]
        },
        {
            "Sid": "SNS",
            "Effect": "Allow",
            "Action": [
                "sns:Publish",
                "sns:SetTopicAttributes"
            ],
            "Resource": "*"
        }
    ]
}
EOF

  create_package         = false
  local_existing_package = data.archive_file.default.output_path
}

resource "aws_lambda_permission" "allow_secret_manager_call_Lambda" {
  count = var.create ? 1 : 0

  function_name = module.lambda_function.this_lambda_function_name
  statement_id = "AllowExecutionSecretManager"
  action = "lambda:InvokeFunction"
  principal = "secretsmanager.amazonaws.com"
}

The lambda code was adapted from here “https://github.com/biswajeetr/User_Key_rotation/blob/master/lambdatodeleteevery90100110days_v2.txt” and now looks like this:

import json
import boto3
import base64
import datetime
import os
from datetime import date
from botocore.exceptions import ClientError
iam = boto3.client('iam')
secretmanager = boto3.client('secretsmanager')
IAM_UserName='${iam_user_name}'
SecretName='${secret_name}'
SNS ='${sns_topic_arn}'

def delete_key():
    try:
        print ("deactivate!")
        getpresecvalue=secretmanager.get_secret_value(SecretId=SecretName,VersionStage='AWSPREVIOUS')
        preSecString = json.loads(getpresecvalue['SecretString'])
        preAccKey=preSecString['access-id']
        iam.update_access_key(AccessKeyId=preAccKey,Status='Inactive',UserName=IAM_UserName)
        print ("delete!")
        keylist=iam.list_access_keys (UserName=IAM_UserName)
        for x in range(2):
            prevkeystatus=keylist['AccessKeyMetadata'][x]['Status']
            preacckeyvalue=keylist['AccessKeyMetadata'][x]['AccessKeyId']
            print (prevkeystatus)
            if prevkeystatus == "Inactive":
                if preAccKey==preacckeyvalue:
                    print (preacckeyvalue)
                    iam.delete_access_key (UserName=IAM_UserName,AccessKeyId=preacckeyvalue)
                    emailmsg="PreviousKey "+preacckeyvalue+" has been deleted for user "+IAM_UserName
                    sns_send_report = boto3.client('sns',region_name='eu-central-1')
                    sns_send_report.publish(TopicArn=SNS, Message=emailmsg, Subject='Previous Key has been deleted')
                    return
                else:
                    print ("secret manager previous value doesn't match with inactive IAM key value")
            else:
                print ("previous key is still active")
        return
    except ClientError as e:
        print (e)

def create_key():
    try:
        print ("create!")
        response = iam.create_access_key(UserName=IAM_UserName)
        AccessKey = response['AccessKey']['AccessKeyId']
        SecretKey = response['AccessKey']['SecretAccessKey']
        getcurrentsecvalue=secretmanager.get_secret_value(SecretId=SecretName,VersionStage='AWSCURRENT')
        loaded_data = json.loads(getcurrentsecvalue['SecretString'])
        loaded_data['access-id'] = AccessKey
        loaded_data['secret-key'] = SecretKey
        json_data=json.dumps(loaded_data)
        secretmanager.put_secret_value(SecretId=SecretName,SecretString=json_data)
        emailmsg="New "+AccessKey+" has been create. Please get the secret key value from secret manager"
        sns_send_report = boto3.client('sns',region_name='eu-central-1')
        sns_send_report.publish(TopicArn=SNS, Message=emailmsg, Subject="New Key created for user "+ IAM_UserName)
    except ClientError as e:
        print (e)

def lambda_handler(event, context):
    try:
        delete_key()
        create_key()
    except ClientError as e:
        print (e)

An overview of the various services, how the credentials get into the K8s cluster and how it is automatically exchanged

If something goes wrong, each technical user has two access keys. So that if the new credentials do not reach K8s, our services still have a functioning access. First the older key will be deleted. Then a new key will be created to replace it in the Secrets Manager. The Kubernetes External Secrets update the K8s Secret. A tool called Reloader updates our services by creating a new replica. After that the new credentials are then used.

stakater/Reloader

Conclusion

Exchanging credentials for technical users can be quite annoying. If a developer accidentally publishes credentials to Git, i press a button and everything is fine again. Unfortunately, not all security information can be changed that easily. As long as everything is in one universe, it works very well. Problems are caused by services that lie outside of that universe. Individual solutions must be created for each service. This is time consuming. Often outdated passwords and a guilty conscience remain.

Connect to Jonas B:

Jonas B on Github