Keeping Your Secrets Secret

Harnessing AWS Secrets Manager and CloudFormation to Automate Secrets Management

The Challenge of Secrets Management

If you haven’t encountered this challenge yet, you will soon: You’re migrating an application to the cloud, or building a new one, and you’ve got a masterpiece of a CI/CD pipeline with just one exception: your secrets. Your database and application passwords, API keys and other credentials are still hard coded and — gasp! — stored in unencrypted config files! So, what’s the best, most efficient method for securely storing, managing and deploying secrets throughout an application’s lifecycle?

This challenge has been well defined and solved for in many different ways. SingleStone’s own Cloud Security Architect, Don Mills, developed a beautiful solution dubbed s3Encrypt which is used in production, at scale, at one of our Fortune 500 clients. You can read a detailed blog post about it here. But, recognizing the need for a native secrets management solution, Amazon Web Services (AWS) released Secrets Manager in April 2018.

AWS Secrets Manager

AWS Secrets Manager enables you to manage, rotate and retrieve database credentials, API keys and other secrets throughout their lifecycle. You can read more about this new offering in the AWS Secrets Manager documentation.

So now you’ve read all about Secrets Manager and you’re excited about it as a simple solution to your secrets management woes. Crestfallen, you realize there are no native CloudFormation Resources to interface with AWS Secrets Manager. I’m sure AWS will eventually announce CloudFormation support for Secrets Manager, but until then, it’s Lambda-backed Custom Resources to the rescue!

Okay, so we’ve defined the challenge: secrets management. We found an elegant solution: AWS Secrets Manager. But now we need to integrate this with our existing CI/CD pipeline, which deploys infrastructure using CloudFormation. Let’s dive in!

Audience and Implementation

This post is intended for cloud and DevOps engineers with at least foundational knowledge of AWS, including the AWS Console, CloudFormation, Lambda functions, and some scripting/programming. We’ll be using Python in the Lambda function, and while you don’t need to be a Python expert (I’m certainly not), you should be able read and follow a basic script.

There’s certainly no shortage of content for creating Lambda-backed Custom Resources, but I’ll focus on the Lambda function itself, which provides the interface to AWS Secrets Manager via the AWS Software Development Kit (SDK) for Python.

We’ll create the following CloudFormation Resources:

• Lambda Function Execution Role: An Identity and Access Management (IAM) Role that grants the Lambda function permissions to access the required AWS resources.
• Lambda Function: The AWS Lambda function definition that includes the API calls to implement the Secrets Manager interface.
• CloudFormation Custom Resources: The CloudFormation Resource that calls our Lambda function.

For the complete Lambda function and CloudFormation template presented here, please refer to the AWS Secrets Manager with CloudFormation Repository.

Structure and Attributes of the Secret

A Secret consists of a version “object” and several metadata elements that describe the secret and define how it should be handled by Secrets Manager. You can read more about these terms here.

Note that in this post we do not implement secret rotation. We are always dealing with the “default” version and don’t specify a Version Tag.

Version Object

Each version that’s maintained for a secret has the following elements:

• A unique ID for the version
• A collection of staging labels that can be used to identify the version (unique within the secret). A version with no staging labels is considered deprecated and subject to deletion by Secrets Manager.
• The secret text that’s encrypted and stored

The encrypted secret text is a JSON object that specifies, at a minimum, the usernameand password:

{
  "username": "myPostgresAdmin",
  "password": "sup3rsecret!@#"
}

Metadata

The Metadata contains the details about the Secret and consists of the following:

• ARN
• Secret Name
• Description
• KMS Key ID
• Rotation Configuration
• Last used date/time stamps
• Tags

Do not confuse the username, which exists in the encrypted secret text, with the Secret Name in the metadata. It’s technically possible for them to have the same value, but I would personally recommend using something different. Additionally, you can use a / in the SecretName to create a “path-like” name that identifies your Secrets by environment, for example: dev/PostgresAdmin.

For more information on the structure and features of Secrets, please read the AWS Secrets Manager User Guide.

Create the Lambda Function Execution Role

When creating a Lambda function, you must specify an Execution Role. At run time, the Lambda function will assume this Role, obtaining the permissions specified in the Role’s attached Policy. You can manually create the Role or create it dynamically using CloudFormation. Since we want the Role to be ephemeral, with a lifecycle attached to our Stack, we’ll use the CloudFormation method.

CloudFormation provides the Resource Type AWS::IAM::Role that can be used to dynamically create an IAM Role. The CloudFormation Resource definition for our Lambda Function Execution Role is below. As always with permissions, it’s best practice to use a “least privilege” model, granting only the absolute minimum permission required for the function to work. This can sometimes be tricky, depending on the complexity of your Lambda function. Start by reviewing the API Reference Guides for the AWS Services and the associated Actions that your function employs. For example, this Lambda function uses the AWS Secrets Manager API get_secret_value method . The API Reference for this command describes the minimum permissions:

To run this command, you must have the following permissions:

secretsmanager:GetSecretValue

kms:Decrypt — required only if you use a customer-managed AWS KMS key to encrypt the secret. You do not need this permission to use the account’s default AWS managed CMK for Secrets Manager.

As such, the secretsmanager:GetSecretValue permission is included in the Policy Actions below. We are not using custom-managed AWS KMS keys, so we do not need kms:Decrypt. By default, Lambda functions will log their output to AWS CloudWatch, so those permissions are also included.

Also note the AssumeRolePolicyDocument Property. This is required, and grants the AWS Lambda Service permission to assume the Role.

Code the Lambda Function

Now comes the fun part: creating the Lambda function itself! Choose your favorite language from the supported runtimes (C#, Go, Java, Node.js, orPython) and use the associated AWS SDK to implement the code that will interface with AWS Secrets Manager. My example uses Python3.6.

This iteration of the Lambda function code provides the following services:

• Retrieve an existing Secret: Retrieve the password for an existing Secret and return it to CloudFormation.
• Upsert a Secret: Create a new Secret or Update an existing Secret with a password provided by the CloudFormation template designer. If you don’t provide a password, Secrets Manager randomly generates one. The password is not returned to CloudFormation.

The Retrieve and Upsert services are implemented using the following AWS Secrets Manager API methods:

• get_secret_value
• put_secret_value
• create_secret
• get_random_password

We will only implement basic functionality in this iteration of function code. We won’t, for example, implement VersionStaging, which allows passwords to be rotated with tracked versions.

For the full function code, please visit the AWS Secrets Manager with CloudFormation Repository.

Lambda Function Handler Objects: Event and Context

Before we dive into explaining the logic and flow of the function, it’s worth spending a moment talking about the Lambda Function Handler event and context objects. These descriptions come from the Lambda Function Handler (Python) Documentation:

event

AWS Lambda uses this parameter to pass in event data to the handler. This parameter is usually of the Python dict type. It can also be list, str, int, float, or NoneType type.

context

AWS Lambda uses this parameter to provide runtime information to your handler. This parameter is of the LambdaContext type.

Currently, our function doesn’t utilize any values from the context object, so we’ll focus solely on the event object.

When CloudFormation triggers the Lambda function, the Lambda Service provides an event object similar to the one below. (In this example, 123456789000 represents your AWS Account Number.)

Let’s examine a couple of these key/value pairs:

• "RequestType": "Create": The value for RequestType is the CloudFormation operation that triggered the Lambda event and can be: Create, Update, or Delete. This allows your function to take different actions on CloudFormation Create, Update, and Delete operations. As currently written, our Secrets Manager Lambda function only acts on Create.
• "ResponseURL": This is an important one. CloudFormation needs to know the exit status of your Lambda function and you may need to pass function output back to CloudFormation. This ResponseURL is an S3 Bucket presigned URL, dynamically created by CloudFormation to receive responses from custom resource providers. This is what you use to tell CloudFormation if your Lambda function status is success or failed. In this case, this is also how we return the Secret Password. See the response object below.
• "ResourceProperties": These are the Properties that you specify in your CloudFormation Custom Resource declaration, which we’ll review later in this post.

Returning a Response

Upon completion, your function must construct a JSON response object and return it to the ResponseURL. Here is an example response object from a successful completion:

• You can see we have returned a Status of “SUCCESS”
• Since the "SecretAction" was get, we need to return the password for the requested Secret. Returning values to CloudFormation is done through the "Data" key. Within the "Data" key, you can specify any keys you want and the values of those keys can be accessed using the CloudFormation Fn::GetAtt intrinsic function. So, back in our CloudFormation template, we can access the value for "SecretPassword" as such:

Fn::GetAtt GetPostgresSecret.SecretPassword

Notice the inclusion of "NoEcho": true. This ensures that values in the "Data" key are not exposed in describe-stack and describe-stack-events output. More on this soon.

Code Logic and Flow

Let’s take a look at the function from the perspective of the two services it provides, Retrieve and Upsert .

Retrieve an Existing Secret

In addition to the standard values provided in the Lambda event object, the Retrieve functionality requires the following parameters:

• SecretName: The Secret Name for which to retrieve the password, which is not to be confused with the username.
• SecretAction: The Action to be taken. In the case of Retrieve, the Action is get.
• Region: Secrets are region specific. Specify the AWS Region from which to Retrieve the Secret.

These Parameters are provided by the CloudFormation template designer in the Custom Resource declaration and are included in the "ResourceProperties" key of the event object. You can see this in the event object example from the previous section. Since we’re only interested in taking action on a CloudFormation Create operation, the function first checks the event['RequestType'].

If the value is Create and the SecretAction is get, the function will call the get_secret function to retrieve the password for the specified SecretName. If the Secret exists, then a Status of "SUCCESS" and the password are returned via the response object as shown in the previous section. If the Secret does NOT exist, then Status of "FAILED" is returned along with the Reason for the failure.

Upsert a Secret

For those who haven’t come across the term “Upsert” before, it’s simply a combination of “update” and “insert”. When you take this action, it will be created if it doesn’t already exist, or updated if it does. In addition to the standard values provided in the Lambda event object, the Upsert functionality accepts the following parameters:

Required

• SecretName: The Secret Name for which to retrieve the password, which is not to be confused with the username.
• SecretAction: The Action to be taken. In the case of Upsert, the Action is upsert.
• SecretUserName: The username for the Secret.
• SecretDescription: AWS Secrets Manager does not require a Secret. Description. However, the Lambda function expects one, so this parameter is required.
• Region: Secrets are region specific. Specify the AWS Region from which to Retrieve the Secret.

Optional

• SecretPassword: The password for the Secret. How could the password be optional? This function provides a little bonus feature. See the explanation below.

These Parameters are provided by the CloudFormation template designer in the Custom Resource declaration and are included in the "ResourceProperties" key of the event object. You can see this in the event object example from the previous section.

Here’s how an optional SecretPassword works: The function checks to see if the SecretPassword parameter was provided. If not, it uses the get_random_password Secrets Manager API method to generate a random password. The get_random_password accepts some optional parameters you can use to control the length, character composition and other properties of the resulting password. However, we’re using straight defaults. You can read more on optional parameters in the GetRandomPassword Documentation.

Similar to the Retrieve Service, the event['RequestType'] is checked. If the value is Create and the SecretAction is upsert, the function will perform the check for the SecretPassword and then call the upsert_secret function. upsert_secret invokes the create_secret method. If there is no error, the Secret has been successfully created and we return a Status of "SUCCESS" along with the Amazon Resource Name (ARN) of the newly created Secret. If we get a ResourceExistsException, then the Secret already exists and the put_secret_value method is invoked to update the Secret with the new password.

A Word on Security

Since we’re dealing with Secrets, I’d be remiss if I didn’t touch on a few important security guidelines for working with AWS Secrets Manager and CloudFormation.

1. Secrets are encrypted at rest and in transit. (See the AWS Secrets Manager User Guide for more detail.)
2. There a few spots in the Secret lifecycle where you need to take precautions to avoid unintentionally exposing password values:

Logging: By default, your Lambda function will log its output to CloudWatch. This is extremely useful for troubleshooting, but if you’re not careful your Secret text could show up here. For example, it’s desirable to log out the Lambda handler event object at the start of your function:

# Log event data to Cloudwatch
logger.info(“***** Received event. Full parameters: {}”.format(json.dumps(event)))

However, if SecretAction is upsert and the template designer provided a password, then that value will be present in the event object as event['ResourceProperties']['SecretPassword']. This value is required in order to store the Secret, but we certainly don’t want to echo it to the log. To prevent this, I included some code that checks for the presence of the event['ResourceProperties']['SecretPassword'] key, extracts the value for later use, and then overwrites it with ******** before logging out the event object.

CloudFormation: There are two places in CloudFormation where Secrets may show up unintentionally:

• Parameters: Always use the NoEcho Property when specifying Secrets as CloudFormation Parameters. This will ensure the Parameter value is masked (*****) whenever the Stack is viewed with the Console or via the describe-stacks or describe-stack-events CLI commands.
• Custom Resource Response Objects: The Response Object also provides a NoEcho Property. Always set this Property to True when working with Secrets to ensure your returned values are masked from the describe-stacks and describe-stack-events output.

Create the Lambda Function

You can create the Lambda function manually via the Console. However, we want this resource to be ephemeral, and linked to the lifecycle of the CloudFormation Stack. So, we’ll define the function dynamically using the AWS::Lambda::Function CloudFormation Resource.

If your function code is short (fewer than 4096 characters), you can specify the code inline using the zipfile: property under the Code: property. See the AWS Lambda Function Code Documentation for more detail. Since our function exceeds the 4096-character limit, we are storing our code package in S3 and using the S3Bucket and S3Key properties to define the code.

Create the CloudFormation Custom Resource

Now that we have defined our Lambda function and granted it appropriate permissions via the Execution Role, it’s time to implement the CloudFormation Custom Resources.

As explained above, in its current version, the Lambda function provides two CloudFormation Custom Resource options:

• Retrieve an existing Secret: Retrieve the password for an existing Secret
• Upsert a Secret: Create a new Secret or Update an existing Secret with a given password. If you don’t provide a password, CloudFormation will randomly generate one.

For more detail, see the Custom Resources section of the AWS CloudFormation Documentation, but here are the important points:

• Use the AWS::CloudFormation::CustomResource or Custom::String resource type to define Custom Resources.
• Custom Resources require one Property: ServiceToken, which specifies the ARN of the AWS resource to which CloudFormation sends the requests. In this case, this is the ARN of our Lambda function.
• You can specify additional Properties and these will be passed into the Lambda function via the ResourceProperties key in the event object. Specifying additional properties is usually optional for Custom Resources; however, your Lambda function may require input parameters, making specification of those Properties required for your Custom Resource. Let’s look at some examples:

1. Retrieve an Existing Secret

For the purposes of this example, you have an existing Secret in AWS Secrets Manager with the name: myPostgresSecret. We would like to retrieve the password for this Secret in our CloudFormation template so that we can pass it to an AWS::RDS::DBInstance resource. To do so, we would create a Custom Resource:

A few things to note:

• As with all CloudFormation Resources, we can name the LogicalId anything we like. Here we’re using GetPostgresSecret
• For Type, you can use AWS::CloudFormation::CustomResource or Custom::String (I.e. Custom::RetrieveSecretPassword)

We have 4 Properties:

• ServiceToken: This Property is required for all Custom Resources and is the ARN of the target Resource. In this case, we’re using the Fn::GetAtt|!GetAtt intrinsic function to get the ARN of the Lamdba function we defined earlier with the SecretsManagerLambdaFunction CloudFormation Resource.
• SecretAction: This Property is required by the Lambda function and specifies the Secret Action (get or upsert).
• SecretName: This Property is required by the Lambda function and specifies the name of the Secret on which to perform the SecretAction.
• Region: This Property is required by the Lambda function and specifies the AWS Region where Secrets Manager commands should be performed. Remember, Secrets are Region specific.

Back in your CloudFormation template, you can then use Fn::GetAtt GetPostgresSecret.SecretPassword (or !GetAtt) to reference the retrieved password for myPostgrsSecret. For example, your AWS::RDS::DBInstance might look like this:

2. Create a New Secret — with a provided password

In this example, you would like to store a new Secret and provide the password. For this, you need to specify a few additional Properties in the Custom Resource:

Let’s look at the fields that differ from the previous example:

• For Type this time, we went with the Custom::String option: Custom::AuroraDBAdminSecret

Creating a new Secret requires 7 Properties:

• ServiceToken: As with example #1, this is the ARN of the Lamdba function as returned by the Fn::GetAtt|!GetAtt intrinsic function
• SecretAction: The Action for creating or updating a Secret is upsert
• SecretName: This is the identifier of the Secret we wish to create
• SecretUserName: To create a new Secret, you need to provide a username. Here we are referencing a CloudFormation template Parameter — dbUserName
• SecretPassword: To create a new Secret, you need to provide a password. Here we are referencing a CloudFormation template Parameter — dbPassword. Be sure to use the NoEcho Property when defining the password Parameter
• SecretDescription: Provide a description for the Secret
• Region: Again, Secrets are Region specific, so specify the AWS Region where this Secret should be created.

3. Create a New Secret — with an auto-generated random password

You can provide greater security for your Secrets by having AWS Secrets Manager generate a random string for your Secret Password. The Properties are exactly the same as Creating a new Secret with a provided password — simply omit the SecretPassword Property:

Similar to Retrieving an Existing Secret, you can reference the randomly generated password via Fn::GetAtt CreateAuroraDBSecret.SecretPassword (or !GetAtt).

Additional AWS Secrets Manager CI/CD Integration

Since our goal is to manage Secrets throughout the CI/CD pipeline, let’s take a moment to look at another common use case for accessing and using Secrets. A simple example might look like this: You have an application consisting of a containerized API that interfaces with a Postgres database. Your application is deployed using two CloudFormation templates: one for the application infrastructure, including the database, and another for your API Docker container.

The infrastructure template uses the resources laid out in this article to create a new AWS Secrets Manager Secret for the database Admin user. The API template accepts a DatabasePassword Parameter that is passed to the container via an Environment variable in the Container Definition. So, in the API deployment stage of your pipeline, you need to provide a value for the DatabasePassword Parameter in the aws cloudformation create-stack command. You can accomplish this by using the AWS CLI to retrieve your database password from AWS Secrets Manager, storing it in a temporary environment variable on your build server, and then using that env var in your create-stack command.

Assuming the following:

• We have a CloudFormation template (not shown) that deploys our Postgres database and creates a DB Admin Secret in Secrets Manager called dev/pg_password.
• We have a CloudFormation template (not shown) that deploys our API to AWS Elastic Container Service and accepts the following Parameters:

Version: The version of the API Container to deploy.
Cluster: The name of the ECS Cluster where the container should be deployed.
EnvironmentName: The environment we are deploying.
DatabasePassword: The password for the Postgres database.

• The build server has JQ installed for parsing json.

Your pipeline could be implemented as follows:

Note: Since your pipeline might be defined in Groovy, YAML, or another language, I’ll simply show the basic commands in bash.

Conclusion

So what I have presented here is a relatively complete and secure solution for creating, managing, and deploying Secrets throughout your application’s continuous integration/continuous deployment lifecycle. Always be sure to carefully consider your organization’s security and compliance requirements when implementing any secrets management solution. As I mentioned, AWS Secrets Manager provides an excellent feature to enable automatic rotation of passwords for a secured service or database, with native support for Aurora, MySQL, and PostgreSQL on RDS. Although the Lamdba function presented here does not implement this feature (yet), it could easily be extended to so for heightened security and to meet compliance requirements.

For the full Lambda function code and a CloudFormation template that implements the Custom Resources, please refer to the AWS Secrets Manager with CloudFormation Repository.