Secrets Management: Using External Secret Operator on GKE and Google Secret Manager.

ESO (External Secret Operator ) — is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information(secrets) from external APIs and automatically injects the values into a Kubernetes Secret.

Introduction

What is the goal of External Secrets Operator?

The goal of External Secrets Operator is to synchronize secrets from external APIs (like GSM) into Kubernetes. ESO is a collection of custom API resources — ExternalSecret, SecretStore and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.

Differnet core resources of External Secret Operator

Provider : Different external Secret manager tools like GCP secret manager,AWS secret manager ,HashiCorp Vault

SecretStore : It define which provider to use and how to authenticate with provider. It’s a namespaced resource

ClusterSecretStore: It is as same as SecretStore but it is cluster-scoped .This type of store can be referenced by all ExternalSecrets from different namespace. Use it to offer a central gateway to your secret backend.

ExternalSecret : The ExternalSecret declares what secret to fetch from external providers. It takes a reference to a SecretStore which knows how to access the provider data.

Google Secret Manager (GSM) — It is GCP provided native service to store ,manage and access secrets securely . Secrets data can any sensitive data like database password,TLS certificates,SSH keys etc.

Secret Manager overview | Secret Manager Documentation | Google Cloud

Kubernetes Secret

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image.

Prerequisite

1. You should have access to a GCP project with permission to create secrets, service account ,bind IAM permission and create GKE cluster

2. Application Deployment Cluster: Set a GKE cluster specifically for deploying the application. In this cluster, we will install External Secret operator to securely access,update and utilize secrets stored in Google Secret Manger .

Steps to execute to install ESO and sync Secrets from Google Secret manager to Kubernetes Secrets

• Setup necessary env variables

export GCP_PROJECT_ID=weighty-legend-415316
export GCP_ZONE=us-central1-a
export ESO_GCP_SERVICE_ACCOUNT=secret-accessor        #  Google IAM Service Account 
export ESO_K8S_NAMESPACE=external-secrets             #  Kubenetes namespace to deploy 
export ESO_K8S_SERVICE_ACCOUNT=external-secrets       #  Kubernetes Service Accounta"

• Create GKE cluster with workload identity with minimum nodes or you can create autopilot cluster using GCP console or gcloud command

 gcloud container clusters create eso-cluster --zone=$GCP_ZONE --workload-pool=$GCP_PROJECT_ID.svc.id.goog --machine-type "e2-medium" --num-nodes "2"  --disk-size "50" \
 --project $GCP_PROJECT_ID --scopes="https://www.googleapis.com/auth/cloud-platform"

• Get credentials for your cluster to authenticate with cluster

gcloud container clusters get-credentials eso-cluster --zone $GCP_ZONE --project $GCP_PROJECT_ID

• Create a namespace to use for the Kubernetes service account. You can also use the default namespace or any existing namespace.

kubectl create namespace app

• Create a service account in GCP and assign IAM permission to access and manage secrets in Google secret manager .

1. create a GCP service account 
 gcloud iam service-accounts create $ESO_GCP_SERVICE_ACCOUNT \
--project=$GCP_PROJECT_ID

2. Assign secret accessor IAM role on service account 

gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \
--member "serviceAccount:$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/secretmanager.secretAccessor" --project=$GCP_PROJECT_ID

• Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.

gcloud iam service-accounts add-iam-policy-binding $ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[$ESO_K8S_NAMESPACE/$ESO_K8S_SERVICE_ACCOUNT]" --project=$GCP_PROJECT_ID

Refer below Google docs to understand above steps to create cluster and configure workload identity to authenticate your workload

Use Workload Identity | Google Kubernetes Engine (GKE) | Google Cloud

Install External Secret Operator (ESO)

• Get credentials for your cluster to authenticate with cluster

gcloud container clusters get-credentials eso-cluster --zone us-central1-a --project deft-sight-402505

• Add the External-Secrets Helm repository

helm repo add external-secrets https://charts.external-secrets.io

• Ensure that Helm is up-to-date by updating all repositories.

helm repo update 

• Install the external-secrets repository using Helm

helm upgrade -install external-secrets external-secrets/external-secrets \
    --set 'serviceAccount.annotations.iam\.gke\.io\/gcp-service-account'="$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
    --namespace external-secrets \
    --create-namespace \
    --debug \
    --wait

. List all objects related to external secret operator

kubectl get all -n external-secrets

• Create a test secret in Google Secret Manager

printf "user1" | gcloud secrets create db-username --data-file=- --project=$GCP_PROJECT_ID
printf "pass1" | gcloud secrets create db-password --data-file=- --project=$GCP_PROJECT_ID

• Now we need to create ClusterSecretStore and External secret

Create ClusterSecretStore Object

cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-store
spec:
  provider:
    gcpsm:
      projectID: $GCP_PROJECT_ID

EOF

Create namespace app for external secret

Kubectl create namespace app

Create ExternalSecret

cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-creds
  namespace: app
spec:
  refreshInterval: 10s            # rate SecretManager pulls GCPSM, Low refereshInternval for demo purpose,Set this value based based on apps
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-store               # name of the ClusterSecretStore or you can also reference SecretStore
  target:
    name: db-creds                # name of the k8s Secret to be created
    creationPolicy: Owner
  data:
  - secretKey: db-user            # name of secretkey it can be any name            
    remoteRef:
      key: db-username            # name of the GCPSM secret key
  - secretKey: db-pass            # name of secretkey it can be any name            
    remoteRef:
      key: db-password            # name of the GCPSM secret key
EOF

• Check Kubernetes secret and it’s data

kubectl get secret db-creds -n app  -o jsonpath='{.data.db-user}' | base64 -d
kubectl get secret db-creds -n app  -o jsonpath='{.data.db-pass}' | base64 -d

• Update the Google manager Secret with new updated value

printf "user2" | gcloud secrets versions add db-username --data-file=- --project=$GCP_PROJECT_ID
printf "pass2" | gcloud secrets versions add db-password --data-file=- --project=$GCP_PROJECT_ID

• Check again Kubernetes secret and it’s data . It should be auto updated because of ESO

kubectl get secret db-creds -n app  -o jsonpath='{.data.db-user}' | base64 -d
kubectl get secret db-creds -n app  -o jsonpath='{.data.db-pass}' | base64 -d

Reference for more details

• Official ESO — https://external-secrets.io/latest/
• Providers- https://external-secrets.io/latest/provider/aws-secrets-manager/
• Kubernetes Secret — https://kubernetes.io/docs/concepts/configuration/secret/
• Using GCP secret Manager — https://external-secrets.io/latest/provider/google-secrets-manager/
• Community Contribution — https://external-secrets.io/latest/contributing/process/