Integrating AWS Secret Manager with EKS and use Secrets inside the Pods: Part-1

This blog provides you enough details on how you can use secrets (managed by AWS Secrets Manager) inside AWS EKS pods.

Overview:

AWS Secrets Manager now enables you to securely retrieve secrets from AWS Secrets Manager for use in your Amazon Elastic Kubernetes Service (Amazon EKS) Kubernetes pods.With AWS Secret and Configuration Provider (ASCP), you can securely store and manage your secrets in Secrets Manager, and retrieve them through your applications that are running on Kubernetes, without the need to write custom code. You also have the added benefit of using AWS Identity and Access Management (IAM) and resource policies on your secret to limit and restrict access to specific Kubernetes pods inside a cluster. This tightly controls which secrets are accessible by which pods. After ASCP is installed and enabled, it helps ensure that your applications always receive the most current version of the secret when the pod starts, enabling you to benefit from the lifecycle management capabilities of Secrets Manager. So, you not only gain the benefit of a natively-integrated secrets management solution, but also the ability to provide configurations in a single provider.

In this post, we will see how to set up AWS Secrets & Configuration Provider (ASCP) to work with the Secrets Store CSI driver on your Kubernetes clusters. The Secrets Store CSI driver allows Kubernetes to mount secrets stored in external secrets stores into the pods as volumes. After the volumes are attached, the data is mounted into the container’s file system.Here,the external secret store is Secrets Manager.

Prerequisites:

This solution has the following prerequisites:

• An AWS account.
• An IAM policy with permissions to retrieve a secret from Secrets Manager.
• Your secret stored in Secrets Manager
• An existing EKS Cluster
• A user that can modify your Kubernetes cluster
• AWS CLI and kubectl installed
• Helm and eksctl installed

Deploying the solution:

This solution includes the following steps, which will be described in more detail in the following sections:

1. Restrict access to your pods using IAM roles for service accounts
2. Install the Kubernetes secrets store CSI driver
3. Install the AWS Secrets & Configuration Provider
4. Create and deploy the SecretProviderClass custom resource
5. Configure and deploy the Pods to mount the volumes based on the configured secrets
6. Load secrets and configurations from the volumes mounted to the container

Brief :In this example, we will see the steps for Simple Nginx deployment to demonstrate how those secrets can be accessed .We will mount the secret as a file as well as expose it to Pod via Environment Variable.

To achieve that we will deploy Kubernetes secrets store CSI driver and AWS Secret Manager and configuration Provider.Working together they will let you retrieve the secret and provide it to the pod.

To configure access, we will create an OpenID connect provider and Associate the IAM Role with the Kubernetes Service Account.

Step 1: Restrict access to your pods using IAM roles for service accounts :

You will use IAM roles for service accounts (IRSA) to limit secret access to your pods. By setting this up, the provider will retrieve the pod identity and exchange this identity for an IAM role. ASCP will then assume the IAM role of the pod and only retrieve secrets from Secrets Manager that the pod is authorized to access.

This prevents the container from accessing secrets that are intended for another container that belongs to another pod.

• IAM OpenID Connect

Copy the OpenID Connect provider URL

Pate the OpenID Connect provider URL and add Auidence

Auidence : sts.amazonaws.com

• Create IAM Policy to read secrets ,here.

Create IAM Policy

• Identity IAM Role for Kubernetes Service Account (IRSA)

Select the OIDC url

Update Trust Relationship (so that only one Kubernetes Service account can use it)

Change aud to sub and change the link (selected)

Step 2: Install the Kubernetes secrets store CSI driver

It integrates secret stores with kubernetes via container storage interface volume.

The Secrets store CSI Driver allows kubernetes to mount multiple secrets ,keys and the volume attached the data is mounted into the container’s file system.

There are two ways of Installation (here) :

1)Deployment using Helm

2)Deployment using YAML

Deployment using Helm

Secrets Store CSI Driver allows users to customize their installation via Helm.

Requirements:

Amazon Elastic Kubernetes Service (EKS) 1.16+

Secrets Store CSI driver installed:

$ helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

$ helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

Note: that older versions of the driver may require the — set grpcSupportedProviders=”aws” flag on the install step.

To install Helm, check here.

Deployment using yamls:

Need to create two Custom Resources Definitions (CRDs)

The CustomResource is an object that extends the kubernetes API or allows you to introduce your own API into a project or cluster. A Custom Resource definition file defines your own object kinds and let the API Server handle the entire lifecycle.

Create secrets-store-csi-driver/0-secretproviderclasses-crd.yaml

Create secrets-store-csi-driver/1-secretproviderclasspodstatuses-crd.yaml

Apply CRDs

$ kubectl apply -f secrets-store.csi.x-k8s.io_secretproviderclasses.yaml

$ kubectl apply -f secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml

The output should show the following Secrets Store CSI driver pods and custom resource definitions (CRDs) deployed:

Create secrets-store-csi-driver/2-service-account.yaml

Create secrets-store-csi-driver/3-cluster-role.yaml

Create secrets-store-csi-driver/4-cluster-role-binding.yaml

#Associate the service account with the RBAC role that we just created

Create secrets-store-csi-driver/5-daemonset.yaml

#To run this secret-store-CSI-Driver on each node created daemonset

Create secrets-store-csi-driver/6-csi-driver.yaml

Apply Kubernetes objects

$ kubectl apply -f secrets-store-csi-driver

Step 3: Install the AWS Secrets & Configuration Provider

The CSI driver allows you to mount your secrets in your EKS Kubernetes pods. To retrieve them from Secrets Manager so the CSI driver can mount them, you need to install the AWS Secrets & Configuration Provider (ASCP).

Create aws-provider-installer/0-service-account.yaml

#Service Account for secret-store-provider

Create aws-provider-installer/1-cluster-role.yaml

Create aws-provider-installer/2-cluster-role-binding.yaml

#Associate the service account with the RBAC role that we just created

Create aws-provider-installer/3-daemonset.yaml

#To run this secret-store-provider on each node created daemonset

Apply aws-provider-installer

$ kubectl apply -f aws-provider-installer

Check Logs:

$ kubectl logs -n kube-system -f -l app=csi-secrets-store-provider-aws

Step 4: Create and deploy the SecretProviderClass custom resource

In order to use the Secrets Store CSI driver, you have to create a SecretProviderClass custom resource. This provides driver configurations and provider-specific parameters to the CSI driver itself.

To use ASCP, you create the SecretProviderClass to provide a few more details of how you are going to retrieve secrets from Secrets Manager. The SecretProviderClass MUST be in the same namespace as the pod referencing it.

• Create secret-provider-class.yaml

The following is an example SecretProviderClass configuration:

Step 5: Configure and deploy the pods to mount the volumes based on the configured secrets

Update your deployment YAML to use the secrets-store.csi.k8s.io driver, and reference the SecretProviderClass resource created previously.

The following is an example of how to configure a pod to mount a volume based on the SecretProviderClass to retrieve secrets from Secrets Manager.

On pod start and restart, the CSI driver will call the provider binary to retrieve the secret and configurations from Secrets Manager and Parameter Store, respectively. After successfully retrieving this information, the CSI driver will mount them to the container’s file system. You can validate that the volume is mounted properly after a restart by running the following command:

#Get a shell into the Container that is running in your Pod:

$ kubectl -n production exec -it nginx-<id> — bash

Print mounted file

cat /mnt/api-token/secret-token

Print environment variables with a secret

echo $API_TOKEN

The complete code for this tutorial is available at : https://github.com/Ishana-HC/AWS-secrets-manager.git

Conclusion:

Secret store Container Storage Interface (CSI) driver enables mounting of secrets, passwords, and certificates from enterprise grade external secret stores as volumes. This project features a pluggable provider interface, which can be leveraged to integrate AWS Secrets Manager as an external secret store for Kubernetes secrets.

***