Managing Application Secrets Like Never Before, Using AWS S3 and CodeDeploy
Managing secrets for your backend application and keeping them secure is a tough matter. In this article, I will discuss how we approached and solved this problem, using the power of CodeDeploy scripts and S3 storage.
CodeDeploy: AWS CodeDeploy is a fully managed deployment service that automates software deployments to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers.
For those of you who don’t know how CodeDeploy works, you basically add an appspec.yml file in the root of your project. This contains something called hooks; each hook represents a life cycle event in the deployment process. So ApplicationStop hook, for example, gets called when the deployment begins, and this hook executes the script under scripts/stop.sh
Note: Keep the AfterInstall hook in mind, as we will be getting our secrets files during the execution of this hook.
Recently, we were working on deploying a proper infrastructure for a nodejs backend application. We had already written the CodeDeploy scripts to stop/start/validate etc. We’d reached the part where we needed to create different deployment environments, and to be able to deploy different secrets and configurations for each environment.
We had the following as given:
- All the secrets exist in .json files under a /conf directory in the root of the application.
- We needed 3 different environments: staging, QA and production.
Each environment will have its own server, along with its own secrets, database urls and API keys etc.
We had already started research on how to manage secrets for different environments, and we had found a couple that were good, but not good enough. We had two goals in mind while working on a solution: flexibility and simplicity. Also, keep in mind that we were following the separation-of-concerns approach; meaning that, we didn’t want the app developers to worry about exporting the secrets on different environments. In addition to that, we didn’t want some script to be edited each time a new secret is added or removed, as this could lead to many hidden bugs due to miscommunication.
In the next section, I will be discussing some solutions we’ve found, some of which we’ve tried and improved. And then I will be explaining our custom solution with the code and the how-to.
Possible solutions:
- Using environment variables:
Cons: I won’t get into details, but this was a controversial approach that had its security concerns. Also, it would’ve been a hassle to automate exporting the variables on different environments, especially that we didn’t want the app developers worrying about exporting them.
Cons:
- Using the AWS SDK in the code to get the secrets through API calls was out of the question. The codebase was already done and reading secrets was something the developer should not worry about. Plus, this will add the headache of caching the secrets on the server so that we don’t make an API call on each access.
- The app developer does not have access to the console. We will need create a CRUD for the secrets.
- The secrets are easily retrievable from the console (anyone with access to the console could see their plain text value). To workaround this, one could store the secrets already encrypted using KMS. This will add an encryption step before saving, and a decryption step after retrieving.
- Cost. It’s not THAT expensive to store a secret (0.4$ per secret per month), but we had ALOT of secrets to store. You could workaround that by storing many secret values into one secret, in a JSON object like so (AWS secrets manager accepts JSON values):
{"secret1":"value1", "secret2":"value2"}but this will increase the complexity when retrieving the secrets.
A good thing about parameter store is that you can save the parameter under a virtual “path”. So for example, for a database key, we had different parameters for different environments saved like this: /staging/dbkey , /prod/dbkey and /qa/dbkey . We went a step further with this, and wrote a short python script that would fill a template of the secrets file, by replacing placeholders on each deployment.
Cons:
- Again, we had A LOT of secrets and configs to store. This caused a mess on parameter store, with all the different secrets and paths depending on the environment. Also, like above, the secrets were easily retrievable from the console.
- Cost. It’s basically free to store standard secrets on parameter store, but you will pay per API interaction with each secret. So for example if you request 15 secrets, you will pay for the retrieval of each. The pricing of this could be negligible, but we don’t control how many secrets we’re gonna store, and we don’t want it to be a problem later on so it’s safer to pass.
- Adding/removing secrets was a MESS. A quick walk through on how this approach worked: Basically we run a CodeDeploy deployment, it fetches the code base from Github, where the conf/ directory exists. Inside the directory, we had a conf.json template file that contains placeholders, something like:
{
"dbkey": {{ db_key }},
"dbpassword": {{ db_password }}}
After the code is downloaded on the server, we execute a script in the AfterInstall hook of CodeDeploy. The script calls parameter store and gets all the parameters for the corresponding environment, then executes another python script to fill the placeholders in the file above.
Now imagine we needed to add a new secret to our file. We will need to add it in the .json template file above, in our our AfterInstall hook script file if we’re filtering on name, and in our python script so that it could find the placeholder and replace it. So that was a big no-no.
Our solution:
Storing the secrets files on a private S3 bucket, structured by environment name; Then retrieving the correct files during deployment, depending on the environment.
This would solve most of the problems we faced above. Let’s walk through how we did it to make the point clearer.
Go to the S3 console, and click on create bucket. After you choose a name for your bucket, there are 2 things you need to make sure of:
- Make sure you choose to Automatically encrypt objects when they are stored in S3. You can choose either of the encryption options.
2. Next, make sure you block ALL public access to the bucket. This step is essential, otherwise your secrets files would become accessible to the internet.
A sample S3 bucket would look like:
And under each environment, prod for example, you will have the desired secrets and config files (JSON files in this case):
Half of the work is already done, really, that simple. All what's left is retrieving the files from the correct bucket upon deploying. To do that, we will add a download_secrets.sh file in our CodeDeploy’s scripts/ directory (this will be called from the AfterInstall hook I previously mentioned). The download_secrets.sh could look something like this:
#!/bin/bashDIR=/home/ubuntu/nodejs_app/conf#create config directory if not exists
mkdir $DIR -p#get secrets files depending on current deployment environment
if [[ "$DEPLOYMENT_GROUP_NAME" == *"STAGING" ]];
then#here we are assuming that the EC2 has a role for accessing the S3 bucket
sudo aws s3 cp s3://secrets-bucket/nodejs-app/staging $DIR --recursive --exclude "*" --include "*.json"elif [[ "$DEPLOYMENT_GROUP_NAME" == *"QA" ]];
thensudo aws s3 cp s3://secrets-bucket/nodejs-app/qa $DIR --recursive --exclude "*" --include "*.json"elif [[ "$DEPLOYMENT_GROUP_NAME" == *"PROD" ]];
thensudo aws s3 cp s3://secrets-bucket/nodejs-app/prod $DIR --recursive --exclude "*" --include "*.json"fi
Explanation:
For those of you who don’t know shell, basically this script figures out which deployment group is running (by reading the $DEPLOYMENT_GROUP_NAME environment variable), and then downloads all the config files (with extension .json in this case) from the corresponding S3 bucket, to the directory we specified ($DIR).
NOTES:
- If you’re not aware what $DEPLOYMENT_GROUP_NAME is, it’s an environment variable that can be accessed from any script during a CodeDeploy deployment. Check it out, there are a couple others that might be of use.
- The script uses the AWS CLI to access and download files from S3. So make sure your instances have it installed.
- Our EC2 instances have an IAM role that has access to our S3 buckets. This should already be set up if you’ve been using CodeDeploy (since CodeDeploy gets the code to deploy from S3).
- When creating our deployment groups in CodeDeploy, we have an agreed upon naming convention; And that is the actual name we want, ending with the environment name. So for example, nodejs_app_DG_PROD. And in the script, we check what the deployment group name ends with in order to figure out which environment we’re currently deploying to.
And that’s it!
This solves most of the issues we’ve faced in the previous solutions.
- There is no mess when storing secrets since the files are put, in a directory, under their environment name. So you will be able to find all secrets for an environment in one place. Also, adding/removing secrets is as simple as the developer editing the file locally (a step which they already do when developing the application), and then uploading the file.
- Almost ZERO cost. S3 is one of the cheapest cloud storages out there, and when you access the files you’re downloading a small amount of data, since the files should be small (even if they weren’t, S3 is still very cheap).
- Security. The buckets are private, so no one could access them from the internet. Also, the files are encrypted at rest AND in transit. In addition to that, it is very easy to manage access to the buckets through IAM policies, roles and permissions. So this flexibility adds a lot of value in my opinion. IF you want an even more secure storage, you can upload the files already encrypted to S3. This, however, will add an extra step for decrypting the files after retrieving them.
You can even take this a step further and create a simple web application that’s internal for your organization, where the secrets files can be uploaded to. This is a useful feature for separating roles of teams (not everyone should have access to the S3 console). It also adds security since not many will be able to read from the S3 console (i.e view the secrets files)
We’ve found this very useful so far, and have in fact taken it a step further. Not only did we add secrets files, but webserver configurations and other config files as well, that vary from one environment to the other.
So if you’re lost or unsure on how to store your secrets and config files, I hope you find this helpful.
On the other hand, if you believe this method is unsecure, or may be breached in some way, then please reach out and let us discuss.
