Integrating AWS Secrets Manager with Kubernetes Using External Secrets Operator

External Secrets

Introduction

Traditionally, secrets, such as API keys, passwords, and certificates, are managed within Kubernetes itself. This approach poses security and operational risks. Storing sensitive secrets alongside application infrastructure increases the risk of exposure. Additionally, manually updating secrets is error-prone and inefficient, particularly in large or dynamic environments.

The External Secrets Operator

The External Secrets Operator mitigates these risks by securely integrating Kubernetes clusters with external secrets management systems, including AWS Secrets Manager, HashiCorp Vault, and Azure Key Vault. This article focuses on leveraging the External Secrets Operator with AWS Secrets Manager to efficiently fetch secrets.

Prerequisites

• AWS Account
• An Amazon EKS cluster
• ArgoCD installed on your cluster for GitOps-style deployment of external secrets operator - documentation
• AWS CLI installed
• KUBECTL installed

1. Create an IAM role:

First, create an IAM role. You’ll be specifying an IAM username to associate with the access key that’ll be used later . Use the following trust policy., replacing <ACCOUNT-ID> and <IAM-USER-NAME> with your details.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT-ID>:user/<IAM-USER-NAME>“
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

2. Create a IAM Policy

Next, create an IAM policy that allows the necessary actions for the External Secrets Operator to interact with AWS Secrets Manager. Replace <ACCOUNT-ID> with your AWS account ID.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:<REGION>:<ACCOUNT-ID>:secret:*"
      ]
    }
  ]
}

3. Install External Secrets Operator

To install the External Secrets Operator, create a file named external-secrets-operator.yml In this file, specify the details of your EKS cluster and the IAM role created earlier. Push this YAML file to the repository from which ArgoCd syncs your deployment manifests.

external-secrets-operator.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: external-secrets
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  sources:
  - repoURL: https://charts.external-secrets.io
    chart: external-secrets
    targetRevision: 0.9.4
    helm:
      values: |         
        replicaCount: 1
        fullnameOverride: "external-secrets"
        clusterName: <cluster name>
        clusterEndpoint: ${dependency.eks.outputs.cluster_endpoint}
        serviceAccount:
         name: external-secrets-operator
         annotations:
          eks.amazonaws.com/role-arn: <IAM ROLE>

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false

4. Create a SecretStore

Before creating a SecretStore, store your AWS access key and secret-key in a Kubernetes secret within your cluster. Create a file named creds.txt replacing <ACCESS_KEY> and <SECRET_ACCESS_KEY> with your actual credentials.

creds.txt

AWS_ACCESS_KEY_ID=<ACCESS_KEY>
AWS_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY>

Then, use the following command to create the Kubernetes secret

kubectl create secret generic awssm-secrets \\
  --from-literal=access-key=$(grep AWS_ACCESS_KEY_ID creds.txt | cut -d= -f2) \\
  --from-literal=secret-access-key=$(grep AWS_SECRET_ACCESS_KEY creds.txt | cut -d= -f2) \\
  --namespace external-secret\

secret-store.yml

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: external-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      role: arn:aws:iam::<ACCOUNT-ID>:role/<IAM ROLE>
      region: eu-west-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

now let’s apply this…

kubectl apply -f secret-store.yml

let’s check if out secrets store has been created and configured.

kubectl get secretstore -n external-secrets

kubectl get secretstore -n external-secrets

after creating the secret store, we’ll now proceed to pull an already saved secret from AWS Secrets Manager.

secrets.yml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: secrets-manager-secret
    creationPolicy: Owner
  data:
    - secretKey: <secret-name>
      remoteRef:
        key: <secret-key>
    - secretKey: <secret-name>
      remoteRef:
        key: <secret-key>

Make sure to replace secret-name with your Secrets Manager Secrets name and secret-key as the name it’ll save it as on your secret store.

The above YML file stores the secret on the target secrets-manager-secret on the namespace external-secrets.

For every 1hr it refreshes and updates the secrets from the SecretStore.

It fetches the species secret from SecretStore aws-secretsmanager and stores it in the target secret secrets-manager-secret.

Use the provided commands to check that your AWS Secrets Manager secret has been successfully retrieved and stored within your Kubernetes cluster

kubectl get secret secrets-manager-secret -n external-secrets

you’ll get the output below

Conclusion

In this guide, we have demonstrated how to automate the management of secrets in Kubernetes using External Secrets Operator and AWS Secrets Manager. This approach does not only enhance the security of secret management within Kubernetes but also simplifies and automates updates making it more efficient and manageable for DevOps teams. Adopting this method ensures that sensitive information is kept out of your application’s infrastructure and is dynamically manageable, scaling seamlessly with your environment’s needs.