A brief introduction to Secret Management

Introduction

The traditional way of locally storing and managing secrets doesn’t fit the modern application nor the scale we have nowadays. Modern applications are highly-available, cloud-based, multi-user, multi-device while the usual user requires reliability, accessibility, and performance.

Fully controlling the security of our application and its dependencies while also leveraging cloud third-party applications is close to impossible. We cannot guarantee the security of our providers nor the security of our app dependencies.

Therefore the need for a way of managing secrets arises. Through this article we will discuss what secrets are, why you need a way to manage them and we will cover the modern model of managing secrets.

What are secrets?

Secrets can be anything that, once exposed, will negatively impact your company. Through this article we will focus on application-specific secrets such as

• Application credentials
• API keys
• TLS certificates
• Connection strings
• Cryptographic keys
• etc…

Why do we need secret management?

Modern architecture has brought us many benefits such as modularity and reliability resulting in high interconnectivity between our services and users. We now have distributed applications that are highly available, composed of hundreds or even thousands of microservices. That alone creates a large footprint and proportionally the possibility of exploitation by malicious software.

As an exercise, imagine you have a server. You may need an ssh key to connect to that server. You may also have a disk encryption key for that server. Probably that server runs an application or multiple applications so you may also need app credentials such as connection string or API keys. For redundancy you may have multiple servers, thus you now have clusters in a datacenter. This application you are serving is also highly-available and now you have multiple data centers scattered across the globe. Not long after and you will end up with managing thousands of keys. This can become tedious and prone to mistakes if you were to manually manage them.

The issue of managing secrets no longer resumes only to keep it secret, we need a solution that needs to be reliable, scalable, secure, and fast. How are we going to control, store, access, and distribute secrets in a cloud environment?

The modern solution

The pattern applied in the modern application is to have a service designated for managing secrets. This model is also known as Secret Storage as a Service(SSaaS). Some of the features we expect from such a service are:

1. Centralized key storage

Having centralized key storage brings us many benefits, including the ability to secure and audit our secrets, manage access via fine-grained policies, rotate secrets, and many more.

2. Access Control

Access control to our secrets should be done via our standard mechanism of authentication and authorization such as OAuth. This will be favorable to SSO systems in turn encouraging users to have only one account. With this pattern, it only becomes reasonable to have stronger passwords.

3. Cryptographic key management

Flexible key management, loosely coupled with the secret storage.

4. Distribution

Since we don’t locally have the keys we need a way to distribute the keys to the service that encrypts/decrypts.

5. Auditing

We must be able to ask and answer questions such as who used this key, when was it used, how was it used.

A working example

In this example, we will be encrypting an environment file using sops with a google KMS key. Afterward, we will setup a github runner to build a docker image, automatically decrypt our secrets and inject it in the build process via docker build arguments.

Technology stack for this example

• Github for secret storage and automation
• Google KMS for key management
• Google IAM for access control
• SOPS for encrypting/decrypting

Bird’s-eye-view of our example

For the sake of simplicity we will not dive deep into subjects such as automatic encryption, rotating secrets or deployment to any cloud.

Setting up our environment

I will be assuming you already have gcloud sdk, sops and a github repository with a Dockerfile that takes build arguments, similar to this one:

FROM python:3.7-alpine
ARG DB_USERNAME=default
ARG DB_PASSWORD=default
ENV DB_USERNAME=$DB_USERNAME
ENV DB_PASSWORD=$DB_PASSWORD
COPY . .
RUN pip install -r requirements.txt
CMD ["python3", "app.py"]

In case you are missing the application but still want to proceed with this example you may fork this repository

Setup google KMS

Create a keyring and a key in google KMS via gcloud for later use by sops.

gcloud kms keyrings create github-sops --location global
gcloud kms keys create sample-secret-key --location global --keyring github-sops --purpose encryption

Retrieve your newly created key

$ gcloud kms keys list --location global --keyring sops
...
projects/<project>/locations/global/keyRings/github-sops/cryptoKeys/sample-secret-key

Setup google IAM

For Github to be able to access the key from google KMS, we must create a service account and add the credentials as Github secrets

• Creating the service account.

gcloud iam service-accounts create github-sample \
    --description="github-sample" \
    --display-name="github-sample"

• Granting access to our service account for using the key.

gcloud kms keys add-iam-policy-binding sample-secret-key \
    --keyring github-sops \
    --location global \
    --member serviceAccount:github-sample@<project>.iam.gserviceaccount.com \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

• Obtaining credentials for our service account.

gcloud iam service-accounts keys create /tmp/github-account.json \
  --iam-account github-sample@<project>.iam.gserviceaccount.com

Registering GCP credentials with GitHub

We will need to authorize Github to access the KMS key so we can decrypt it with sops.

For this step, you will have to go to your Github UI, in your project settings, and access the secrets section. There you will have to register the following secrets

• GCP_PROJECT get this from GCP console
• GCP_KEY obtained earlier, stored in /tmp/github-account.json
• GCP_EMAIL github-sample@<project_id>.iam.gserviceaccount.com

Creating and encrypting secrets

Let’s create an environment file containing two variables. We will store this file in secrets/env.

DB_USERNAME=secretdbusername
DB_PASSWORD=secretdbpassword

Encrypting our env file and removing it as is no longer needed in plaintext version.

sops --encrypt --gcp-kms projects/<project>/locations/global/keyRings/github-sops/cryptoKeys/sample-secret-key secrets/env > env.enc
rm secrets/env

Creating a GitHub action

This example should serve as a starting point for decrypting secrets with github and nothing more. You should decide how you build, publish and deploy your application.

Having the following Github action, we can automatically decrypt our secrets, inject them during build time, and publish our image.

name: Setup, Build, Publish and Deploy
on: 
  push:
    branches:
      - master
# Environment variables available to all jobs and steps in this workflow
env:
  GCP_PROJECT: ${{ secrets.GCP_PROJECT }}
  GCP_EMAIL: ${{ secrets.GCP_EMAIL }}
  GITHUB_SHA: ${{ github.sha }}
  IMAGE: sample-secret-management
  GOOGLE_APPLICATION_CREDENTIALS: /tmp/github-account.json
  REGISTRY_HOSTNAME: docker.io
  DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
jobs:
  setup-build-publish-deploy:
    name: Setup, Build, Publish, and Deploy
    runs-on: ubuntu-latest
    steps:
- name: Checkout
      uses: actions/checkout@v2
# Setup gcloud CLI
    - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '270.0.0'
        service_account_email: ${{ secrets.GCP_EMAIL }}
        service_account_key: ${{ secrets.GCP_KEY }}
# Download sops and install
    - name: Setup sops
      run: |
        sudo wget https://github.com/mozilla/sops/releases/download/v3.6.0/sops-v3.6.0.linux -O sops
        sudo chmod 0755 sops
        sudo mv sops /bin
        # Configure default gcp authentication
        cat > /tmp/github-account.json <<EOL
        ${{ secrets.GCP_KEY }}
        EOL
        gcloud config set project $GCP_PROJECT
      
    # Build the Docker image
    - name: Build
      run: |  
        # Decrypt secrets
        decrypted_secrets="$(sops --decrypt secrets/env.enc)"
        # Parse decrypted env secrets into docker args format
        build_args="$(for i in $decrypted_secrets;do echo --build-arg $i;done)"
        # Building image and redirecting output to /dev/null to mask our secrets
        docker build $build_args \
          -t $REGISTRY_HOSTNAME/$DOCKER_USERNAME/$IMAGE:$GITHUB_SHA \
          -t $REGISTRY_HOSTNAME/$DOCKER_USERNAME/$IMAGE:latest . > /dev/null
# Push the Docker image 
    - name: Publish
      run: |
        docker login -u $DOCKER_USERNAME -p ${{ secrets.DOCKER_PASSWORD }}
        docker push $REGISTRY_HOSTNAME/$DOCKER_USERNAME/$IMAGE
- name: Deploy
      run: |
        echo "your deploy code here"

Running our example

For validating this example I exposed the secrets at localhost:8000/

Running and testing our sample:

docker pull docker.io/stefanm88/sample-secret-management
docker run -d -p 8000:8000 docker.io/stefanm88/sample-secret-management
curl localhost:8000
    DB_USERNAME=secretdbusername
    DB_PASSWORD=secretdbpassword

Conclusion

We saw that we can use GitHub, SOPS, and GCP as our SSaaS facilitator. Although this solution may be complex, it has some strengths and weaknesses over the secret manager offered by cloud providers such as AWS and GCP.

Strengths:

• Separation of encrypted data from its keys
• Minimal trust between parties
• Modularity and Flexibility
• Secrets can stay with the code

Weaknesses:

• Much more complex to implement and maintain
• Not as many features out-of-the-box

Even though this solution does not have as many features as a cloud secret manager, it does have many opportunities such as rotating secrets via Github cronjobs, auto-encryption with Github hooks, and many more.

The pattern used here is optimal if you want to have the secrets with the source code, and don’t completely trust your cloud provider to manage the secrets for you.

The entire code can be viewed at:

stefanm8/sample-secret-management

References:

Cloud External Key Manager | Cloud KMS Documentation | Google Cloud

mozilla/sops