RDS secrets rotation and ECS update

As every solution comes from a challenge or problem. Engineers love to solve problems and problem-solving requires creativity and knowledge of the tools you have available to you, very similar to an artist that want to express their art with different materials and tools that they have. So, here is the challenge/requirement solved:

Challenge

Database secrets rotation can be a compliance requirement or simply to enhance the security of your environments. Rotate RDS passwords in AWS is facilitated using the Secrets Manager service. However, you may have a fully automated deployment in your infrastructure as ECS blue-green deployments and want to refresh secrets also in your application when a new secret is created.

So, it is a challenge that can be tackled using AWS Lambda functions and here is how you can do it in an efficient and secure way on AWS.

Solution

Every solution starts with an efficient and solid architecture, the language chosen was Python due to its synchronous nature and vast source of libraries as for example boto3, a very popular and well-maintained set of functions to interact with AWS services. Below you can check a High-Level Diagram of the solution with its sequential steps.

1. The Secret Rotation Lambda function connects to the RDS using the current secret stored in SSM parameters store.
2. Generates a 32 characters random password including ascii_letters, digits and 4. punctuation (excluding “:/@“\’\“) as those are separators characters as required by PostgreSQL
3. If current secret successfully connects, then rotates the secret in the database
4. Test new secret before update application parameters
5. If success, then the lambda builds the database URL and RDS secret.
6. Then, finally restarts the application containers on ECS to refresh secret

Disclaimer: Keep in mind that this solution has some downtime and can be triggered during long term basis outside of business hours for example. If you cannot afford any downtime at all, you can use your application to manage the database connection pool and grab the secrets from SSM or Secrets Manager instead of using environment variables into ECS containers.

So, let’s get our hands dirty and deep drive into our Lambda function!

Let’s check some code ⌨️

For start, let’s import our dependencies, this solution imports the PyGreSQL classic interface and pgdb compliant module for PygreSQL written by Ch. Zwerschke and D’Arcy J.M. Cain respectively. Also, it imports typical libraries for string manipulation, timing and logging, for example.

And then our lambda function handler, here you can associate the steps 1 to 6 described before. Due to the synchronous process required for this solution, we can clearly see step by step how the python script will execute safely the secret rotation.

The generate_secret function has some particulates, as PostgreSQL does not accept certain punctuations in the password, be aware when generating a new secret, here is how the current function works:

As in this scenario, the ECS service leverages SSM parameters store to build the tasks, restarting the application containers is necessary.

Don’t forget to create a Cloudwatch trigger for your Lambda function and keep in mind that it is not a zero-downtime approach, chose the cron expression wisely.

🧙

The End

If you read until this point and want to implement this solution on your AWS environment, I do not want to take more of your time and let you know that you can refer to the Github repository below, feel free to contribute or fork it anytime!

wvxavier/ecs_rds_secret_rotation

At DNX Solutions we bring a better cloud experience for digital-native companies in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, Service Mesh, Data Engineering & Analytics, and Managed Services.

Check our open-source projects at https://github.com/DNXLabs and follow us on Linkedin.