Automatic password rotation 🔐

How we no longer have to worry about security

J B Man
J B Man
Jan 25 · 5 min read

“PASSWORDS ARE LIKE UNDERWEAR: YOU DON’T LET PEOPLE SEE IT, YOU SHOULD CHANGE IT VERY OFTEN, AND YOU SHOULDN’T SHARE IT WITH STRANGERS“ (Chris Pirillo).

“What if you could change underwear automatically?” is what went through my head when someone criticises the red exclamation marks in AWS Identity and Access Management (IAM).

snapshot of the access key age before and after the automatic password rotation

Setting up auto-rotation of credentials in K8s on AWS

Our services run in containers in Kubernetes (K8s). We use AWS Elastic Kubernetes Service (EKS) as the service to manage this. Our universe of services and tools also includes AWS IAM to manage our technical users and AWS Secrets Manager to manage all the credentials for our services. To get the credentials from AWS Secrets Manager to K8s we use a tool called Kubernetes External Secrets.

The technical user has security credentials (access key ID and secret access key). With these credentials every technical user has corresponding permissions. For example: We use one technical user to read and write objects in one specific AWS S3 bucket. The given permissions are only for this specific S3 bucket and the user can not do anything else. If we create the user with Terraform, we store the credentials in a new secret inside the Secrets Manager:

resource "aws_iam_user" "default" {
name = "${var.environment}-${var.team_name}-${var.bucket_name}-s3-user"
}
resource "aws_iam_access_key" "default" {
user = aws_iam_user.default.name
}
resource "aws_secretsmanager_secret" "secret" {
name = "${var.environment}/${var.bucket_name}-s3"
description = "This is a terraform generate secret for the s3 bucket ${var.bucket_name}. ATTENTION: DO NOT MODIFIED BY HAND."
}

resource "aws_secretsmanager_secret_version" "secret_value" {
secret_id = aws_secretsmanager_secret.secret.id
secret_string = jsonencode(map("access-id", aws_iam_access_key.default.id, "secret-key", aws_iam_access_key.default.secret))
}

Our services are defined by Helm Charts and deployed in the K8s cluster by Flux. An external secret yaml is defined in the Helm Chart, so that the Kubernetes External Secrets service knows which secret it has to pack where:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: {{ .Chart.Name }}-secrets
namespace: {{ .Release.Namespace }}
spec:
backendType: secretsManager
dataFrom:
- {{ .Values.environment }}/binary-leancms-s3

Our service found the K8s Secret with the credentials and thereby gets access to the AWS S3 bucket. All of our access to AWS resources work according to this principle:

Universe of tools and services
Universe of tools and services
An overview of the various services and how the credentials get into the K8s cluster

Activating auto rotation of the security credentials

The AWS Secrets Manager can configure to automatically rotate the secrets. To do this, it calls a specified lambda for a defined cycle. In our case every 30 days. The code for the lambda in this case, is provided by us. The lambda has permission for the technical user and the corresponding secret inside the Secrets Manager.

resource "aws_secretsmanager_secret_rotation" "default" {
secret_id = aws_secretsmanager_secret.secret.id
rotation_lambda_arn = module.iam-secret-rotate-lambda.arn

rotation_rules {
automatically_after_days = 30
}
}

For the lambda code, i wrote a little Terraform module, which in turn uses the official Terraform AWS lambda module.

data "archive_file" "default" {

type = "zip"
output_path = "${path.module}/src/package.zip"

source {
content = templatefile("${path.module}/src/index.tpl", { iam_user_name = var.iam_user_name, secret_name = var.secret_name, sns_topic_arn = var.sns_topic_arn})
filename = "index.py"
}
}

module "lambda_function" {
source = "terraform-aws-modules/lambda/aws"
version = "v1.24.0"

create = var.create
function_name = "${var.environment}-${var.team_name}-${var.service_name}-iam-secret-rotate-lambda"
description = "A function to rotate IAM Access Keys and Secrets Manager Secrets"
handler = "index.lambda_handler"
runtime = "python3.8"
create_role = true
attach_policy_json = true
policy_json = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SecretManager",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:PutSecretValue",
"secretsmanager:DeleteSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:UpdateSecret",
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:RestoreSecret",
"secretsmanager:UpdateSecretVersionStage",
"secretsmanager:RotateSecret"
],
"Resource": [
"${var.secret_arn}"
]
},
{
"Sid": "IAM",
"Effect": "Allow",
"Action": [
"iam:DeleteAccessKey",
"iam:UpdateUser",
"iam:GetAccessKeyLastUsed",
"iam:UpdateAccessKey",
"iam:CreateAccessKey",
"iam:GetUser",
"iam:ChangePassword",
"iam:ListAccessKeys"
],
"Resource": [
"arn:aws:iam::${var.account_id}:user/${var.iam_user_name}"
]
},
{
"Sid": "SNS",
"Effect": "Allow",
"Action": [
"sns:Publish",
"sns:SetTopicAttributes"
],
"Resource": "*"
}
]
}
EOF

create_package = false
local_existing_package = data.archive_file.default.output_path
}

resource "aws_lambda_permission" "allow_secret_manager_call_Lambda" {
count = var.create ? 1 : 0

function_name = module.lambda_function.this_lambda_function_name
statement_id = "AllowExecutionSecretManager"
action = "lambda:InvokeFunction"
principal = "secretsmanager.amazonaws.com"
}

The lambda code was adapted from here “https://github.com/biswajeetr/User_Key_rotation/blob/master/lambdatodeleteevery90100110days_v2.txt” and now looks like this:

import json
import boto3
import base64
import datetime
import os
from datetime import date
from botocore.exceptions import ClientError
iam = boto3.client('iam')
secretmanager = boto3.client('secretsmanager')
IAM_UserName='${iam_user_name}'
SecretName='${secret_name}'
SNS ='${sns_topic_arn}'

def delete_key():
try:
print ("deactivate!")
getpresecvalue=secretmanager.get_secret_value(SecretId=SecretName,VersionStage='AWSPREVIOUS')
preSecString = json.loads(getpresecvalue['SecretString'])
preAccKey=preSecString['access-id']
iam.update_access_key(AccessKeyId=preAccKey,Status='Inactive',UserName=IAM_UserName)
print ("delete!")
keylist=iam.list_access_keys (UserName=IAM_UserName)
for x in range(2):
prevkeystatus=keylist['AccessKeyMetadata'][x]['Status']
preacckeyvalue=keylist['AccessKeyMetadata'][x]['AccessKeyId']
print (prevkeystatus)
if prevkeystatus == "Inactive":
if preAccKey==preacckeyvalue:
print (preacckeyvalue)
iam.delete_access_key (UserName=IAM_UserName,AccessKeyId=preacckeyvalue)
emailmsg="PreviousKey "+preacckeyvalue+" has been deleted for user "+IAM_UserName
sns_send_report = boto3.client('sns',region_name='eu-central-1')
sns_send_report.publish(TopicArn=SNS, Message=emailmsg, Subject='Previous Key has been deleted')
return
else:
print ("secret manager previous value doesn't match with inactive IAM key value")
else:
print ("previous key is still active")
return
except ClientError as e:
print (e)

def create_key():
try:
print ("create!")
response = iam.create_access_key(UserName=IAM_UserName)
AccessKey = response['AccessKey']['AccessKeyId']
SecretKey = response['AccessKey']['SecretAccessKey']
getcurrentsecvalue=secretmanager.get_secret_value(SecretId=SecretName,VersionStage='AWSCURRENT')
loaded_data = json.loads(getcurrentsecvalue['SecretString'])
loaded_data['access-id'] = AccessKey
loaded_data['secret-key'] = SecretKey
json_data=json.dumps(loaded_data)
secretmanager.put_secret_value(SecretId=SecretName,SecretString=json_data)
emailmsg="New "+AccessKey+" has been create. Please get the secret key value from secret manager"
sns_send_report = boto3.client('sns',region_name='eu-central-1')
sns_send_report.publish(TopicArn=SNS, Message=emailmsg, Subject="New Key created for user "+ IAM_UserName)
except ClientError as e:
print (e)

def lambda_handler(event, context):
try:
delete_key()
create_key()
except ClientError as e:
print (e)
An overview of the various services, how the credentials get into the K8s cluster and how it is automatically exchanged

If something goes wrong, each technical user has two access keys. So that if the new credentials do not reach K8s, our services still have a functioning access. First the older key will be deleted. Then a new key will be created to replace it in the Secrets Manager. The Kubernetes External Secrets update the K8s Secret. A tool called Reloader updates our services by creating a new replica. After that the new credentials are then used.

Conclusion

Exchanging credentials for technical users can be quite annoying. If a developer accidentally publishes credentials to Git, i press a button and everything is fine again. Unfortunately, not all security information can be changed that easily. As long as everything is in one universe, it works very well. Problems are caused by services that lie outside of that universe. Individual solutions must be created for each service. This is time consuming. Often outdated passwords and a guilty conscience remain.

Connect to Jonas B:

Jonas B on Github

Axel Springer Tech

Tech and product folks writing about their work across the…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store