External Secrets Operator: A Cloud Native Secret Management Solution

6 min readJul 14, 2023

In today's cloud-native landscape, managing secrets is crucial for ensuring the security and integrity of applications and infrastructure. Secrets such as API keys, database credentials, and encryption keys need to be securely stored and accessed by applications without exposing sensitive information.

This is where the External Secrets Operator comes into play, offering a robust and efficient solution for cloud-native secret management.

External Secrets Operator(ESO)

External Secrets Operator is a Kubernetes operator that helps to manage secrets in a secure and scalable way. The operator synchronizes secrets from external secret management systems, such as HashiCorp Vault, AWS Secrets Manager, and Google Secret Manager, into Kubernetes secrets. This allows you to store secrets in a secure, centralized location and then access them from your applications in a secure and controlled way.

Managing secrets in Kubernetes can be a pain for several reasons, but fortunately, the External Secrets Operator provides solutions to alleviate these challenges. Here's why managing secrets in Kubernetes can be painful and how the External Secrets Operator helps:

Security Risks: Kubernetes secrets are stored in plaintext within the cluster, which poses a security risk. The External Secrets Operator addresses this by allowing you to store secrets in external secret stores, such as HashiCorp Vault or AWS Secrets Manager, which provide more robust security features like encryption, access controls, and audit trails. By leveraging external secret stores, the operator ensures sensitive information is securely stored and managed. Use RBAC to control the access to Kubernetes secrets and enable etcd encryption.
Manual Management: Managing secrets manually in Kubernetes can be error-prone and time-consuming. The External Secrets Operator automates secret management by fetching secrets from external secret stores and injecting them into applications dynamically. This eliminates the need for manual intervention, reducing the risk of misconfiguration and saving time for administrators and developers.
Secret Distribution: Distributing secrets securely to applications running in Kubernetes can be challenging. The External Secrets Operator simplifies this process by seamlessly integrating with external secret stores and handling the secure retrieval and injection of secrets into the application's environment variables or volumes. This eliminates the need to expose or pass secrets in configuration files through insecure channels.
Secret Rotation: Regularly rotating secrets is crucial for maintaining security, but it can be a complex task in Kubernetes. The External Secrets Operator provides automated secret rotation capabilities by integrating with external secret stores that support key rotation. This ensures secrets are regularly updated without requiring manual intervention or application redeployment, improving security and reducing operational burden.
Cross-Cluster and Multi-Cloud Challenges: Managing secrets consistently across multiple Kubernetes clusters or hybrid/multicloud environments can be challenging. The External Secrets Operator offers a unified interface for managing secrets across various clusters and cloud providers. It integrates with different secret stores, ensuring a consistent and centralized approach to secret management, regardless of the underlying infrastructure.
Auditing and Compliance: Achieving auditability and compliance when managing secrets in Kubernetes is critical. The External Secrets Operator integrates with external secret stores that provide robust auditing and compliance features. This ensures that access to secrets is tracked, audit trails are maintained, and compliance requirements are met, simplifying the process of demonstrating adherence to regulatory standards.

Architecture

The External Secrets Operator extends Kubernetes with Custom Resources, which define where secrets live and how to synchronize them. The operator's controller retrieves secrets from an external API and generates corresponding Kubernetes secrets. Whenever the external API's secret undergoes a change, the controller automatically reconciles the state within the Kubernetes cluster and updates the related secrets accordingly.

Core Resources

Provider: A provider in External Secrets Operator (ESO) is a Kubernetes custom resource that defines how ESO should interact with an external secret management system.
SecretStore: The provider details are configured using SecretStore which is namespaced. The SecretStore maps to exactly one instance of an external API.
ClusterSecretStore: The ClusterSecretStore is a cluster-scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces. Use it to offer a central gateway to your secret backend.
ExternalSecret: The ExternalSecret describes what data should be fetched, how the data should be transformed and saved as a Kind=Secret

Refer to the API guide for ESO architecture details.

Let's see External Secrets Operator(ESO) in action

In this blog post, I will showcase the integration of the External Secrets Operator with a GKE Cluster, specifically focusing on fetching secrets from Google Cloud Secrets Manager.

Set up the necessary env variables.

export GCP_PROJECT_ID="your-project-name"
export ZONE="gcp-zone" #ex: europe-west2-a
export ESO_GCP_SERVICE_ACCOUNT=external-secrets
export ESO_K8S_NAMESPACE=external-secrets
export ESO_K8S_SERVICE_ACCOUNT=external-secrets

Create a GKE cluster with workload identity enabled. Skip this step if you are using an existing cluster.

gcloud container clusters create eso-demo \
   --workload-pool=$GCP_PROJECT_ID.svc.id.goog \
   --zone $ZONE \
   --project $GCP_PROJECT_ID \

Set up workload identity resources for ESO.

#Create GCP service account
gcloud iam service-accounts create $ESO_GCP_SERVICE_ACCOUNT \
--project=$GCP_PROJECT_ID

#Create IAM role bindings
gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \
--member "serviceAccount:$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/secretmanager.secretAccessor"

#Allow kubernetes service account to impersonate GCP service account
gcloud iam service-accounts add-iam-policy-binding $ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[$ESO_K8S_NAMESPACE/$ESO_K8S_SERVICE_ACCOUNT]"

Install External Secrets Operator(ESO).

helm repo add external-secrets https://charts.external-secrets.io

helm upgrade -install external-secrets external-secrets/external-secrets \
    --set 'serviceAccount.annotations.iam\.gke\.io\/gcp-service-account'="$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
    --namespace external-secrets \
    --create-namespace \
    --debug \
    --wait

Set up a sample secret in GCP secret manager.

printf "name=eso-demo-v1" | gcloud secrets create eso-demo --data-file=- --project=$GCP_PROJECT_ID

Setup ClusterSecretStore for GCP secret manager.

cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-store
spec:
  provider:
    gcpsm:
      projectID: $GCP_PROJECT_ID
EOF

ESO is now configured to consume the secrets stored in GCP secret manager. Let's create a ExternalSecret to fetch the data from GCP secret manager and create a kubernetes secret.

cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: eso-demo-credential
  namespace: default
spec:
  refreshInterval: 10s             # rate SecretManager pulls GCPSM, Low refereshInternval for demo purpose,Set this value based on your requirement to limit the number of API calls to GCP Secret Manager.
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-store               # name of the ClusterSecretStore (or kind specified)
  target:
    name: eso-demo-credential    # name of the k8s Secret to be created
    creationPolicy: Owner
  data:
  - secretKey: name
    remoteRef:
      key: eso-demo      # name of the GCPSM secret key
EOF

ESO will fetch the target secret from GCP secret manager and create a kubernetes secret based on the configuration.

Let's add a new version to the existing GCP secret.

printf "name=eso-demo-v2" | gcloud secrets versions add eso-demo --data-file=- --project=$GCP_PROJECT_ID

From the above screenshot, it is apparent that ESO automatically synchronised the new secret version from GCP Secrets Manager, eliminating the need for recreating the Kubernetes secret. You can now incorporate the Kubernetes secrets in pod definitions as environment variables or volumes.

Conclusion

The External Secrets Operator provides a robust solution for managing secrets in a cloud-native environment, guaranteeing security, efficiency, and compliance in secret management processes. By utilizing ESO, organizations can streamline secret management, improve application security, and simplify the administration of secrets in Kubernetes. This blog post aims to shed light on the advantages of adopting ESO and offers a guide to help you get started.