Secret Manager: Improve Cloud Run security without changing the code

Credentials to a database, API keys or private key (for TLS certificate for example) are the types of data that most of applications required to work. These data are secret and lesser people know them, stronger is your secret!

However, I saw, and I continue to see too many times secrets in plain text in repositories or in deployment script.

Why the developer don’t keep the secrets secured?

Sadly, before the release of Secret Manager, there were no managed solution for storing securely the secrets and they were stored in plain text in the deployment scripts and thus is the deployed Cloud Run services.
And, still today, this bad practice is kept by the developer. The teams often know that’s bad, but they didn’t know (and maybe still don’t know) how to do better.

Environment variable usage

Because you have several environments, the secret’s values are different according with the environment. Indeed, you won’t use the Prod credential for your Dev database.

Every developer knows that these values have to be dynamic and not hardcoded (I hope!). One of the best way to do this is to use environment variable. And the secrets are put in plain text in them.

Secret Manager keep your secrets, secret

In December 2019, Secret Manager has been released in public Beta, and 4 months later in GA. Secret Manager allows you to store several version of a secret and only the account with the correct IAM role can access or update them.

The essential is performed: you never set plain text values in environment variables, only a reference to the secret and you code load it at runtime!

You can use gcloud command to manage the secrets, or you can use client libraries for a programmatic use in your code.

Secret Manager and legacy code

Secret Manager is the solution to use in any new development. However, for the existing application you have to update the code to use Secret Manager. This can be an issue.

In addition, some applications rely on the environment variable at startup, and adding a decryption step is impossible or add a huge overhead, especially in legacy application.

Transparent use with Cloud Run

The power of Cloud Run and the container is the capability to customize the running environment.

The idea is to retrieve the secrets and to load the secret value into the environment variable before starting the application. I created a script which perform this

The requirements

Before going deeper into the script, the existing secrets set in the environment variable values have to be created into Secret Manager. You can use gcloud command for this

echo "my super secret" | gcloud beta secrets create \
  --data-file=- --replication-policy=automatic my-secret

Note: the dash - in --data-file param get the result of the previous command, here the echo of the secret

Then, you have to change the environment variable values with this pattern

<prefix>[<procjectID>]/<secretName>[#<version>]

• <prefix> flags the environment variable as having to recovered into Secret Manager. By default, the script use the value secret:
• <projectID> is optional. It defines in which project is stored the secret. If missing, the current project is used. The slash / is always required
• <secretName> is the name of the secret into Secret Manager
• #<version> is optional. It defines the version of the secret into Secret Manager. If missing, the latest version is recovered.

In our test we will use this value secret:/my_secret#1

Access to Secret Manager

The service account used by Cloud Run must be able to reach the secrets into Secret Manager. I recommend you to create a specific service account for each Cloud Run service. The service account will be able to access to secrets and having a fine grained authorization is a best practice.

Let’s start by creating a service account

gcloud iam service-accounts create cr-access-secret

Then, you can either grant the service account at project scope for accessing the secrets. It’s not recommended if all secrets aren’t for this service account

gcloud projects add-iam-policy-binding \
--member=serviceAccount:cr-access-secret@<PROJECT_ID>.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor <PROJECT_ID>

Replace <PROJECT_ID> with your current project ID

Or you can grant the service account on only one specific secret. It’s a repetitive action if you have several secrets, but the security is higher.

gcloud beta secrets add-iam-policy-binding  \
--member=serviceAccount:cr-access-secret@<PROJECT_ID>.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor my-secret

Replace <PROJECT_ID> with your current project ID

Closer look to the Dockerfile

Now, the context is set for the test. If you have a look to the Dockerfile, you will see these 2 last lines

RUN wget https://storage.googleapis.com/secret-loader/start.sh \
     && chmod +x /start.sh
CMD ["/start.sh", "/server"]

I simply download a bash script and I add the execution permission on it. Then I start this script, with only one parameter: what I want to run after the secrets loading.

The secret loading

All the process is performed by the start.sh script file. Let’s go deeper in it.

I start by defining the <prefix> default value. Change it if you want!

secret_prefix="secret:"

Then, the processing is very simple

• Scan all the environment variables that contain the <prefix> and extract the key and value of each

for i in $(printenv | grep ${secret_prefix})
do
  key=$(echo ${i} | cut -d'=' -f 1)
  val=$(echo ${i} | cut -d'=' -f 2-)

• Then I check if the value start with the <prefix>. If so, I remove the <prefix>. Indeed another environment variable could match the grep but without starting by the prefix.

if [[ ${val} == ${secret_prefix}* ]]
then
  val=$(echo ${val} | sed -e "s/${secret_prefix}//g")

• Extract the project, and if it not empty, prepare the parameter value for the gcloud command

projectId=$(echo ${val} | cut -d'/' -f 1)
secret=$(echo ${val} | cut -d'/' -f 2)

if [[ -n ${projectId} ]]
then
  project="--project=${projectId}"
fi

• Extract the secret’s name and its version. If the version is missing, the value latest is used

secretName=$(echo ${secret} | cut -d'#' -f 1)
version="latest"
if [[ ${val} == *#* ]]
then
  version=$(echo ${val} | cut -d'#' -f 2)
fi

• Finally, get the secret and load it into the environment variable with the key name

plain="$(gcloud beta secrets versions access --secret=${secretName} ${version} ${project})"
#For multiline management
export $key="$(echo $plain | sed -e 's/\n//g')"

That’s all. After the loop over the environment variable, I call the only one parameter of the script.

#run the following command
${1}

It’s an unique parameter. In the Dockerfile set the whole command and its arguments into the same string

Build container and deploy the Cloud Run service

For testing the correct loading of the secrets, I wrote a very simple Go server which displays all the environment variables when you call it. Build the container, for example with Cloud Build

gcloud builds submit -t gcr.io/<PROJECT_ID>/secret-loader

Replace <PROJECT_ID> with your current project ID

And deploy the service, with the secret environment variable and the authorized service account

gcloud run deploy --image=gcr.io/<PROJECT_ID>/secret-loader \
--platform=managed --region=us-central1 --allow-unauthenticated \
--service-account=cr-access-secret@<PROJECT_ID>.iam.gserviceaccount.com \
--set-env-vars=super_secret=secret:/my-secret#1 secret-loader

Replace <PROJECT_ID> with your current project ID. For easier tests, I allowed unauthenticated user. Not mandatory, do what suits you best!

Finally, test your deployment either by clicking on the URL provided after the deployment, or by command line

curl https://secret-loader.<project-hash>-uc.a.run.app

Transparent loading

Now you have it. By using this pre-launch script, the secrets are loaded into your environment variables. Your legacy applications don’t have to be updated, only the Docker file has changed, only 2 lines in the best case.

The usage is not limited to Cloud Run and containers. You can also use this pre-launch script in Compute Engine.

Refactorisation, complexity, delay won’t be no longer an excuse to let an application not secured. Keeping your secrets secret is paramount!

Feel free to update, adapt, change the script according with your requirements. I will be happy to know in the comments your usages and your updates.

Update April 18th 2020

Spring Cloud GCP solution

For those that use Spring Cloud, there is a built-in solution. The spring-cloud-gcp project offers a large library for integrating seamlessly the Google Cloud product.

One of them is dedicated to Secret Manager and allows you to easily load your secrets simply by configuration in 3 easy steps

• Add the dependency to your project

With Maven

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-secretmanager</artifactId>
</dependency>

With gradle

dependencies {   compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-secretmanager' }

• Activate the bootstraping at startup in the bootstrap.properties file, and define a prefix for your secrets (recommended)

# Enable the bootstrap at startup
spring.cloud.gcp.secretmanager.bootstrap.enabled=true

# Optional prefix of the secret name
spring.cloud.gcp.secretmanager.secret-name-prefix=secrets.

• Define your secrets directly into the properties file, without changing your code

my_super_secret=${secrets.my_secret}

my_secret is the secret name into secret manager, with the defined prefix, and my_super_secret is the name of the variable that you can inject in your code like this:

@Value(${my_super_secret})
private String mySuperSecret

If you prefer updating your code, or code with this library, you can directly use the secret like this (I don’t recommend this, you hardcode the name of the secret into Secret Manager directly in your code)

@Value(${secrets.my_secret})
private String mySuperSecret

That’s all, no code required, only configuration and variable redefinition in the properties file! More examples in the sample file