Securing Kubernetes Secrets: Integrating AWS Secrets Manager with EKS

In a previous blog post titled “Secure Secrets using Azure Key Vault with Helm Chart,” I received numerous inquiries regarding the integration of Azure Key Vault with AWS EKS for AWS cloud users. Many questioned whether it would be feasible or if there is an alternative service available in AWS for secret storage.
The good news is, you can definitely achieve this integration. In fact, it’s one of several methods for managing secrets in Kubernetes.
In this blog, you will delve into the topic of securing secrets by integrating AWS Secrets Manager, a powerful secret storage service provided by AWS, with EKS.

Before we proceed with the implementation, let’s ensure that we have all the necessary prerequisites in place:

1. AWS account: Make sure you have an active AWS account that you can use to access the required services.

2. IAM policy: Create an IAM policy with appropriate permissions to retrieve a secret from Secrets Manager. This policy should grant the necessary access to read secrets stored in Secrets Manager.

3. Secret stored in Secrets Manager: Ensure that you have already stored the secret you want to use in Secrets Manager. If you haven’t done so yet, create a secret in Secrets Manager with the relevant credentials or sensitive information that you need to secure.

4. Existing EKS Cluster: You should have an existing EKS cluster set up in your AWS account. If you haven’t created one yet, refer to the AWS documentation for instructions on setting up an EKS cluster.

5. User with Kubernetes cluster modification access: Make sure you have a user or an IAM role with the necessary permissions to modify your Kubernetes cluster. This user should be able to deploy configurations, update secrets, and make changes to the cluster as needed.

6. AWS CLI and kubectl: Ensure that you have the AWS CLI (Command Line Interface) and kubectl (Kubernetes control tool) installed on your local machine. These tools will be used to interact with AWS services and manage your Kubernetes cluster.

Once you have fulfilled these prerequisites, you can move forward with the implementation and explore the integration of AWS Secrets Manager with EKS.

To enable the storage and retrieval of secrets stored in AWS Secrets Manager as files mounted in Kubernetes pods, we can utilize the AWS provider for the Secrets Store CSI Driver. This integration involves the use of the AWS Secrets and Configuration Provider (ASCP) to manage the secrets stored in Secrets Manager and make them accessible to workloads running on Amazon EKS.

To get started, follow the steps outlined below to install the Secret Store CSI Driver and ASCP:

Step 1. Install the Secrets Store CSI Driver:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver - set enableSecretRotation=true - set syncSecret.enabled=true - version 1.2.4

Step 2. Install ASCP

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

By executing these commands, you will install the Secrets Store CSI Driver and ASCP into your Kubernetes cluster, enabling the retrieval of secrets from Secrets Manager within your EKS pods.

To create and mount a secret in an EKS pod, follow the steps below:

1. Set the AWS Region and the name of your cluster as shell variables:

REGION=us-west-2
CLUSTERNAME=test-cluster

2. Create a secret and take note of the <SECRETARN>:

aws --region "$REGION" secretsmanager create-secret --name test-secret --secret-string '{"username":"rajith", "password":"Q7VRTDyM4qRZ"}'

3. Create a resource policy for the pod that limits its access to the previously created secret. Replace <SECRETARN> with the ARN of the secret. Save the policy ARN in a shell variable:

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name eks-deployment-policy --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
            "Resource": ["<SECRETARN>"]
        }
    ]
}')

4. If you don’t already have an IAM OIDC provider for the cluster, create one using the eksctl command:

eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once

5. Create the service account that the pod will use and link it to the resource policy created in step 3. For this tutorial, let’s use the service account name eks-deployment-sa:

eksctl create iamserviceaccount --name eks-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

By following these steps, you will create a secret, establish a resource policy for the pod to access the secret, and create a service account associated with the policy. This will allow you to mount and utilize the secret within your EKS pods securely.

To create and mount a secret in an EKS pod, follow the remaining steps below:

1. Create the SecretProviderClass manifest, named Example-SecretProviderClass.yaml, which specifies the secret to be mounted in the pod. Make sure the objectName and objectAlias match the secret's keys.

Example-SecretProviderClass.yaml:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: eks-test-secret
spec:
  provider: aws
  secretObjects:
  - secretName: test-secret
    type: Opaque
    data: 
    - objectName: username 
      key: username
    - objectName: password 
      key: password
  parameters:
    objects: |
        - objectName: "test-secret"
          objectType: "secretsmanager"
          jmesPath:
            - path: "username"
              objectAlias: "username"            
            - path: "password"
              objectAlias: "password"

Apply the SecretProviderClass manifest:

kubectl apply -f Example-SecretProviderClass.yaml

1. Create a deployment manifest file, named ExampleDeployment.yaml, to deploy an Nginx pod. The secret will be mounted at the path /mnt/secrets-store.

ExampleDeployment.yaml:

kind: Service
apiVersion: v1
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: eks-deployment-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "eks-test-secret"
      containers:
      - name: nginx-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
        env:
        - name: username
          valueFrom:
            secretKeyRef:
              name: test-secret 
              key: username
        - name: password
          valueFrom:
            secretKeyRef:
              name: test-secret
              key: password

Apply the deployment manifest:

kubectl apply -f ExampleDeployment.yaml

After applying the manifest, you can retrieve the AWS Secret’s content from the environment variables username and password using the following commands:

kubectl exec -it <POD_NAME> -- env | grep username
kubectl exec -it <POD_NAME> -- env | grep password

Replace <POD_NAME> with the actual name of the pod created by the deployment.

By following these steps, you will successfully create and mount a secret in an EKS pod, allowing you to access the secret’s content through environment variables within the pod.

I hope you found this blog post helpful. Thank you for reading!