Sync Kubernetes secrets with cloud secret managers

When we deploy an application on Kubernetes cluster, We may need to set admin credentials for the application through environment variable which is set using Kubernetes secret.

Its good practice to rotate the credentials periodically. Cloud-native secret managers like AWS Secret manager, GCP secret manager and other provide the feature of secret rotation. This feature can be utilized to rotate Kubernetes secret if we are able to sync Kubernetes secret with cloud-native secret manager.

External secret operator is a Kubernetes operator which make this job easy. It supports management of Kubernetes secret using cloud-native secret managers.

In its own words from its documentation,

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

In this article, We will have a small demo to explore and understand External Secret Operator.

Sync secret maanagers with Kubernetes secret

Here, I will be using Google Secret manager on GCP and sync it to my Kubernetes cluster running on my Virtual Box.

Contents

1. Why it works on my machine?
2. External Secret Operator — CRDs
3. Install External Secret Operator on cluster
4. Demo — Create secret in GCP secret manager
5. Demo — Sync Kubernetes secret with GCP secret manager
6. Demo —Trigger manual sync
7. Demo — Sync specific type of secret (Example: TLS)

Why it works on my machine?

I am using:

• Kubernetes version 1.28
• One control plane and two worker nodes on Ubuntu 22.04
• Containerd CRI version 1.6.25
• Kubectl v1.27 and helm v3.13.1
• Ubuntu 23.10 as Jumpbox to connect to cluster

Cluster info

CRDs

There are three important CRDs in External Secret Operator — SecretStore , ClusterSecretStore and ExternalSecret.

1. SecretStore: This is CRD created by External Secret Operator using which we can define “How to access the cloud secret manager”. With this we can create a custom resource of kind: SecretStore where we define credentials for accessing cloud secret manager. It is namespaced resource and can be referenced from same namespace for use.
2. ClusterSecretStore: It has same function as SecretStore with Cluster-wide scope. It can be referenced from any namespace for use.
3. ExternalSecret: This is CRD created by External Secret Operator using which we can define “What to fetch from cloud secret maanager”. With this we can create a custom resource of kind: ExternalSecret where we define “What K8s secret to be created” and “From where to fetch the secret value”.

Install External Secret Operator

To install External Secret Operator, We need helm.

1. Add helm repository

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

2. Install External Secret Operator

helm install external-secrets \
   external-secrets/external-secrets \
   -n external-secrets \
   --create-namespace

3. Once all its pods are up and ready. We are ready to start our demo.

helm release

Install External Secret Operator

External Secrets pods

Demo

Create secret in GCP secret manager

We need to create a secret in cloud secret manager. I am using GCP secret manager for this demo.

If you have Google Kubernetes Engine (GKE) on Google cloud, Workload identity can be used instead of service accounts to authenticate. Here, I will use service account key since my cluster is on-prem.

1. Create a service account in Google cloud

Go to IAM & Admins → Service Accounts

Create a service account

2. Generate private key for the service account and save it safe and secure in your jumpbox.

Open manage keys

Create new key

Choose JSON as key type

3. Create a secret in secret manager

Open Secret manager

Create a secret and add a value

4. Grant the “secret access” permission to the service account we created before

Click on listed secret

Click on Grant Access from Permissions tab

Grant Secret Manager Secret Accessor role to service account

Sync Kubernetes secret with GCP secret manager

Now that we have secret in cloud secret manager. Lets see how we can sync it to our Kubernetes secret.

I will be performing this demo in Kubernetes namespace demo.

1. Create a Kubernetes secret using the json key of service account we had generated.

kubectl -n demo create secret generic gcp-cred --from-file="secret-access-credentials=</path/to/service-account-key.json>"

Create Kubernetes secret for cloud credential

2. Create SecretStore which will be used by External Secret Operator to authenticate to Google cloud. Below is the example used in this demo:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-store
  namespace: demo
spec:
  provider:
    gcpsm:                                  # <------------------------ Secret manager provide. 'gcpsm' for GCP. For AWS, use 'aws'
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: gcp-cred                  # <------------------------ Kubernetes secret where Service account credential is stored
            key: secret-access-credentials  # <------------------------ Kubernetes secret key which stores credential value
      projectID: <google-project-id>        # <------------------------ Project id of GCP under which secret is created

Create secret store

3. Create ExternalSecret which will create and sync Kubernetes secret with Google secret manager. Below is the example used in this demo:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: appcred
  namespace: demo
spec:
  refreshInterval: 5m             # <---------------- Sync frequency
  secretStoreRef:
    kind: SecretStore             # <---------------- SecretStore or ClusterSecretStore
    name: gcp-store               # <---------------- Name of SecretStore to use for authenticating
  target:
    name: appcred                 # <---------------- Name of the k8s Secret to be created
    creationPolicy: Owner
  data:                           # <---------------- Details of data to be stored in created Kubernetes secret
  - secretKey: ADMIN_PASSWORD     # <---------------- Kubernetes secret key in data block
    remoteRef:
      key: mypassword             # <---------------- Name of GCP secret to fetch the value of key

Create ExternalSecret

4. Once External Secret is created. It will fetch the secret value from secret manager and create Kubernetes secret in cluster. You can check the secret created and the value.

K8s secret is created by External Secret Operator

5. Now lets, Rotate the value of secret in Google secret manager and wait for it to sync to K8s cluster.

Update the value in secret manager

6. Wait for 5 min (Our set refreshInterval). It should sync the value to cluster.

New synced value

Trigger manual sync

We can also trigger the sync manually.

Lets first change the value again in secret manager.

New value of secret in secret manager

Now, lets trigger the manual sync. It is done by annotating ExternalSecret with force-sync=<current-epoch-time>.

kubectl -n demo annotate externalsecret appcred force-sync=$(date +%s) --overwrite

Check the value of secret after adding above annotation. It should have recent value from secret manager.

Manual sync of secret

Sync Specific type of secret

In above example, It will by default create secret of generic or opaque type. But we can also create and sync secret of specific type. Lets create a secret of type TLS here.

1. Create a secret manager which store TLS certificate and provide Secret manager secret accessor to service account.

I have generated a dummy key and certificate and created new secrets in secret manager.

Certificate in Google Secret manager

private key in Google secret manager

2. Same as before, Lets create ExternalSecret. This time we will need to add template field to it. Below example has been used in this demo.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: tls-cert
  namespace: demo
spec:
  refreshInterval: 5m             # <---------------- Sync frequency
  secretStoreRef:
    kind: SecretStore             # <---------------- SecretStore or ClusterSecretStore
    name: gcp-store               # <---------------- Name of Secret store to used for authenticating
  target:
    name: tls-certificate         # <---------------- Name of the k8s Secret to be created
    creationPolicy: Owner
    template:                     # <---------------- Template of K8s secret to be created
      type: kubernetes.io/tls     # <---------------- Type of Kubernetes secret
      data:
        tls.crt: "{{ .mytls }}"   # <---------------- TLS certificate value. This value is fetched from .spec.data[0] in same YAML
        tls.key: "{{ .mykey }}"   # <---------------- TLS private key value. This value is fetched from .spec.data[1] in same YAML
  data:
  - secretKey: mytls              # .spec.data[0]   
    remoteRef:                    
      key: tls-cert               # <---------------- Name of GCP secret in which certificate is stored
  - secretKey: mykey              # .spec.data[1]
    remoteRef:
      key: tls-key                # <---------------- Name of GCP secret in which private key is stored

Create ExternalSecret for tls type of Kubernetes secret

3. Lets check the secret, If its created and synced.

tls.crt value from TLS K8s secret

tls.key value from TLS k8s secret

Yes. Now we can see Kubernetes secret of TLS type created in cluster.

External Secret Operator can be helpful in rotating passwords for application deployed on cluster or to rotate the TLS certificate after it expires, or to enable GitOps by syncing it to GitLab variables.

I hope you found this article helpful. For more details on External Secret Operator, You may read its documentation at https://external-secrets.io/latest/

You can also support my work by buying me a cup of coffee on https://www.buymeacoffee.com/linuxshots

Thank You!

Navratan Lal Gupta

Linux Shots