Using Secrets Manager to manage your secrets
Do you have secrets? Keep them safe!
We at Affinity Answers are now using AWS Secrets manager to manage our secrets and we gave a talk at the AWS Users Group Bangalore on 20th August 2022. As a followup to the live demo at the talk, here is a detailed post on how one can use AWS Secrets Manager.
The Problem
The above is a sample architecture of medium.com (we do not know much about this portal’s architecture, but a calculated guess!), when a user types medium.com using it internet the user connects to the servers in medium.com, for simplification let’s assume there is a cache and a database. Now for the web app to access this maybe through some kind of API, the API needs some kind of credentials to connect to the cache or DB, no? That is the kind of secret we are talking about.
What is a secret?
A secret is something that allows you to connect securely to your application, e.g. connection details to a database, auth service, cache or any other service.
The Challenges
- Hard-coding secrets in your code
It should be very clear to us that this is not a good idea.Anytime the database connection detail changes, we need to modify the code, commit it, build it and deploy it again. The development and operations are combined not a good idea.We got this Database Credentials for sample application from here.
Maybe a slightly better idea would be store the connection details in some kind of configuration files, now at-least you can connect to DB without changing your code, but still your config file should be committed, built and deployed again.
Those who code should not know the “code”
Normally one of developers job is to write code which can connect to the service, given correct credentials are provided. The credentials are provided by operations or admin. So here we are trying to make sure the developers know there is a way to get secrets, without them knowing exactly what it is at any given point in time.
Managing multiple secrets.
In the medium application regarding database we were talking about database being accessed by API for a real world user, database might also be accessed by some BI application internally, should the secret be replicated here, if so how to mange if secrets changes, pull the secret in multiple places etc.
We need a centralized place where we can access the secret and here comes AWS Secrets manager to the rescue.
AWS Secrets Manager
AWS Secrets Manager provides a way to securely store and access your secrets, you can modify the secret centrally. This allows you to separate development and operations like you want. It also allows you to automatically rotate the secrets using lambda function, more on this later.
Demo Application
This is a sample code in Python connecting to a MySQL database from a config file.
You can run the code like this
Now code and config are together in source control, let’s start from here and make these two independent of source control.
Using AWS Secrets Manager to Secure our Secret
Before we start with doing something practical, make sure you have an AWS account with access to Secrets Manager, for creating a secret and AWS IAM to restrict access to that secret
Creating a Secret
Search for Secrets Manager in your console and Click on Store a new secret.
In the above case the secret is the config file, so we will copy the contents of it in plain text mode, in Key, value the secret will be stored in JSON which we don’t need. We choose other type of secret as our secret doesn’t belong to any of this.
Next we need to name our secret AWS suggests the following naming convention {env}/{application}/{secret_name}, you can name however you want we felt this is good and went ahead with this, so we named our secret as dev/MyApp/db , implying that our secret is for development, app is named MyApp and the secret is for db.
In Step 3: you will notice something called rotation, we will re-visit it after sometime but for now let it be as it is and continue.
That’s it!, you have created your first secret.
Accessing your Secret
Now that we have a secrets let’s try accessing it, you can access from the console, click on the Secret you created and you will see Retrieve the Secret, and it should reveal your secret.
You can edit the secret if you wish to and it will be live almost instantly, solving one of the problems of duplicate secrets we talked about, now all the applications which need secrets can listen from here.
You can get the secret using any language AWS has SDK for, like Java, Python, Node etc. This link can help you with that. For now we will use AWS CLI.
To get the secret you only need to know the Secret ARN, which you can easily get from the console or list-secrets API
aws secretsmanager get-secret-value --secret-id {secret_arn}Note: replace the {secret_arn} with the one you have for your secret
The output is in JSON and the secret is available in SecretString Key, let’s extract this using JQ tool allowing us to extract what we need from the JSON using command line
aws secretsmanager get-secret-value --secret-id {secret_arn} | jq -r .SecretStringThis should give an output like the following
This is the secret we saved!, now we can delete the conf.ini from our git repo and instead get it like this.
so the running command would change to the following.
Step 1: Download the secrets from AWS Secrets Manager to the config file
aws secretsmanager get-secret-value --secret-id {secret_arn} | jq -r .SecretString > conf.iniStep 2: Connect to the MySQL db using this secret like before
python3 mysql_connection_test.py conf.iniThat’s it now our demo application is using secrets manager.
Restricting access to secret
Now let’s make sure that the team which deals this secret has access to this. With the required permission, this can be done using AWS IAM.
Let’s see our User Structure in AWS IAM and then we can add permissions from there.
The TeamStark is the one team we want to give access to this secret, create a policy for this group we have named the policy as project_team_stark_secrets_access , this will allow the team to access read/write the secrets.
In this case we are giving the team read/write permissions, feel free to give according to you team requirements.
In Resources choose specific resources and click on Add ARN, you should see this, paste your Secret ARN.
Add, Review and Save. Now your team should be able to access this secret. Your secret is now accessible by the team and admin users.
Rotating the Secrets
“Treat your password like your toothbrush. Don’t let anybody else use it, and get a new one every six months.” — Clifford Stoll
It is a good security practice to change your secret often, to avoid secret leaks and therefore preventing data leaks. AWS suggests us to change the secret as often as 30 days. Changing/updating secret is called as rotating the secret.
The architecture
Before our application used to connect to MySQL (service) using the config file in repository.
Now
- First get the secret from Secrets Manager to application
- Connect to the service (MySQL) using the secret.
- The access to secret is restricted using AWS IAM.
- AWS Secrets Manager based on your provided interval in days can call your AWS lambda function to rotate the secrets automatically.
Let’s talk about the lambda function to rotate your secret in a bit more detail.
- Create Secret
Before you change your secret, you need to know what your next secret is going to be. e.g. : before changing your password, you need to know what your new password is going to be. Your lambda function should be able to generate a new secret.
- Set Secret
The lambda function should be able to update the secret in your service. In case of DB it should have capability to update the password.
- Test Secret
Once you have updated your secret you should be able to validate that the lambda function is able to access the service with the new secret as expected. If it fails here your production applications may not be able to access this service either, be sure to rollback if that happens.
- Finish Secret
After the secret is changed in service it’s time to update AWS Secrets Manager with the new secret, now all the existing applications can use the new secret.
If you are using one of the existing AWS Service like AWS RDS, AWS Document Store etc, you are in luck AWS has the template for Lambda and can be accessed from here. This can also be used to understand on how to write lambda function for your own service.
Strategies for Rotating Secrets
Based on your use case, AWS has two different strategies to rotate for you.
Single User Rotation Strategy
This works if you are using a single user in your application which has permissions to change their password/secret.
This works well for Automated testing of your front-end application where you use a single user to login, or DB where the user can updated their password etc.
- Connect to service using the users credentials secret.
- Generate a new secret/password for the service.
- Change the current secret/password using the connection.
- Test to make sure everything looks good, rollback otherwise.
- Update the new secret in Secrets Manager so that all other applications depending on this secret can access it.
There is an issue here, it occurs at step 3.
The secret is changed in service first and then updated in secrets manager. During this time applications trying to connect will be using the old secret leading to authentication error. To solve this issue AWS suggests implementing something like this on your application end. In simple words, retry connecting to the service after sometime of sleep, hopefully you should be able to connect quickly in a few tries.
There is also a case when during your testing let’s say it failed and you decide to rollback it also fails, the application will not be able to access the service. This would be a very unlikely case, but do make sure your lambda function is rigorously tested.
Alternating Users Rotation Strategy
The second strategy for rotating the secrets, works in case your architecture is like this.
In this strategy, there will be a super user who can do anything you can imagine the service does, for DB it could be creating tables, dropping tables, creating users, deleting users, update user permissions etc. If at all you are using this for Application please change it, it could be a security hazard.
Instead there is something called an app user, this app user will have exactly the permissions the app needs, nothing more nothing less.
In this case of situation AWS suggests Alternating User Strategy.
In this strategy, the lambda clones the App User and creates a clone in this case named App User Cloned. The cloned user will have same permissions as the app user. In this strategy the secret of only one user is changed in a given rotation. So let’s say you have decided to rotate the secret every 30 days, in the first rotation say 30 days later secret of App user is changed, 2nd rotation (60 days) later the secret of App User cloned is changed, so on and so forth. The secret is rotated in an alternating fashion, hence the name alternating user strategy.
Notice the green boxes, so only the first and second box changes. Instead of using the app user we connect to service using the super user, then change the secret of App user in an alternating manner.
More on this can be read from AWS Docs here.
Conclusion
Like we saw AWS Secrets Manager can be used to securely save and access the secrets. It provides a way to restrict access to secret, the rotating user feature if used correctly should make the infrastructure lot more secure.
That’s all folks, hope you got to learn about AWS Secrets Manager and you use it to keep your infrastructure safe. Would like to know how you all use it and what you feel about this!.
The lambda template samples by AWS can be found here in Github.
