Integrating AWS Secrets Manager in Amazon Elastic Kubernetes Service.

In this post we are going to see how we can integrate AWS Secrets Manager[1] with Amazon EKS[2].

We are going to use AWS Secrets And Configuration Provider (ASCP) for Kubernetes Secrets Store CSI Driver[3] which will fetch Secrets from AWS Secrets Manager and will present them as files mounted in EKS Pods.

To describe which Secret to fetch from Secrets Manager have to create a SecretProviderClass[4] YAML file. The Provider Class should be in the same namespace as the Amazon EKS pod it references.

Installing ASCP 💻

To install the Secrets Store CSI Driver and ASCP by using Helm use the following commands.

# Secrets Store CSI Driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true

# syncSecret.enabled=true is a must as this will ensure our secrets are in sync with the mounted secrets from AWS Secrets Manager.

secrets-store-csi-driver Pods in kube-system namespace (k9s[5])

# ASCP
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

secrets-provider Pods in kube-system namespace (k9s[5])

Set up access control 🔐

To grant your Amazon EKS pod access to secrets in Secrets Manager. You first have to create an IAM role for service account and attach the SecretsManagerReadWrite policy to it.

# Creating IAM Service Account
eksctl create iamserviceaccount --name <SERVICE_ACCOUNT_NAME> --namespace <NAMESPACE --cluster <CLUSTER_NAME>--attach-policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite --profile=<PROFILE_NAME> --approve

# Replace SERVICE_ACCOUNT_NAME, NAMESPACE, CLUSTER_NAME & PROFILE with your actual values.

service account named secret-manager-sa created.

Creating SecretProviderClass YAML 📃

# SecretProviderClass.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: "<NAME>"
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: <SECRET_NAME_IN_AWS_SECRETS_MANAGER>
          objectType: secretsmanager
          objectAlias: <ALIAS_OF_SECRET>
  secretObjects: # this will create secrets object                               
  - secretName: "<SECRET_NAME>"
    type: Opaque
    labels:
      secretTag: "<SECRET_TAG>"
    data:
    - objectName: "<OBJECT_NAME>"
      key: "<KEY_NAME>"

# NAME: whatever name you want to provide as metadata
# SECRET_NAME_IN_AWS_SECRETS_MANAGER: name of secret in AWS Secrets Manager that you want to fetch.
# ALIAS_OF_SECRET: this is optional if you want to give any alias to you secret
# SECRET_NAME: the of the secrets Object that is going to be created in EKS.
# OBJECT_NAME: ALIAS_NAME or objectName provider under spec.
# KEY_NAME: under which key you secrets will be present inside secrets Object.

# Switch to namespace where your Pod belongs to and run
kubectl apply -f SecretProviderClass.yaml

# this will create a service provider class

secretproviderclass created.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: {{.Values.application.name}}
  name: {{.Values.application.name}}
spec:
  ...
  template:
    ...
    spec:
      serviceAccountName: <NAME_OF_SERVICE_ACCOUNT_CREATED_EARLIER>
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "SECRET_PROVIDER_CLASS_NAME"
      ...
      containers:
      - name: APP_NAME
        image: <IMAGE>
        env:
          # Name of the secret in secretProviderClass.yaml
          - name: APPLICATION_SECRETS
        valueFrom:
          secretKeyRef:
            # these values will be same as provided in secretProviderClass.yaml
            name: "<SECRET_NAME>"
            key: "<KEY_NAME>"
      ...
      volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      ...

#Deploying App
kubectl apply -f deployment.yaml

In above deployment.yaml we have used the secrets-store.csi driver to get secrets from our Secret Provider Class.

After deploying the app by adding above mentioned volume and mounting that volume to the container. Every Pod is going to have secrets present under the directory /mnt/secrets-store/ with filename being object_name/alias_name.

Also since we have provided an env to our pod by mentioning it in deployment.yaml which is fetching its value from the the secret object that we created earlier using our SecretsProviderClass.yaml. In our application code we can get our secrets in following ways.

// Example in Node.JS

import dotenv from 'dotenv';

// APPLICATION_SECRETS IN env name as exposed through deployment.yaml.
const config = JSON.parse(process.env.APPLICATION_SECRETS);
console.log(config.DB_USERNAME, config.DB_PASSWORD);

// directly getting the credentials from the path itself.
dotenv.config({ path: '/mnt/secrets-store'});

How Secrets are mounted to Pods.

References 🔗

1. AWS Secrets Manager — https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html
2. Amazon EKS — https://aws.amazon.com/eks/
3. Secrets Store CSI Driver — https://secrets-store-csi-driver.sigs.k8s.io/
4. SecretProviderClass — https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver_SecretProviderClass.html
5. K9s — https://k9scli.io/
6. Helm — https://helm.sh/

Follow me: LinkedIn | Twitter (X) | Github