Azure Key Vault provider for Secrets Store CSI Driver in an Azure Kubernetes Service (AKS)

Azure Key Vault, a service offered by Azure for securely storing and managing sensitive data, is pivotal in cloud environments like Azure Kubernetes Service (AKS) where applications require secure access to sensitive information. To address this need, an tool named Secrets Store CSI Driver facilitates access to sensitive data stored in Azure Key Vault for containerized applications running on AKS clusters. This article will delve into how Azure Key Vault provider for Secrets Store CSI Driver is utilized in AKS clusters, the security measures it provides, and the advantages it offers for applications.

Security measures provided:

Centralized management of sensitive data: Azure Key Vault enables centralized management of sensitive data, ensuring that data is accessed and managed from a single trusted source, thus meeting security and compliance requirements.

Data encryption: Azure Key Vault encrypts data at rest, ensuring access control and protection against unauthorized access.

Access controls and monitoring: Azure Key Vault provides access controls and monitoring capabilities, allowing tracking and auditing of which users or applications can access which data.

Advantages for applications:

Secure data storage and access: Azure Key Vault Provider for Secrets Store CSI Driver facilitates secure access to sensitive data for applications, thereby enhancing data security.

Easy integration: This solution enables seamless integration of applications running on AKS clusters with Azure Key Vault, reducing concerns for application developers regarding data security.

Ensuring compliance: Azure Key Vault complies with numerous compliance standards and fulfills data security requirements, assisting applications in meeting compliance requirements.

Azure Key Vault Provider for Secrets Store CSI Driver provides a robust tool for secure data management in AKS clusters, assisting applications in meeting their data security requirements. This allows both developers and business owners to focus on data security.

You can review the link for Azure Key Vault basic concepts at https://learn.microsoft.com/en-us/azure/key-vault/general/basic-concepts, and the link for Azure security baseline for Key Vault at https://learn.microsoft.com/en-us/security/benchmark/azure/baselines/key-vault-security-baseline.

In this article, we’ll discuss how to securely use sensitive information stored in Azure Key Vault within your Azure Kubernetes Cluster (AKS). Specifically, we’ll explore how to store sensitive data like database usernames and passwords in Azure Key Vault and securely utilize them in the AKS environment. We’ll cover the best practices for accomplishing this task and address potential challenges that may arise during implementation. Additionally, we’ll delve into the Secret Store CSI Driver, a critical component in this process, enabling you to store your sensitive information securely and utilize it as environment variables in AKS.

By leveraging Azure Key Vault and the Secret Store CSI Driver, you can ensure the confidentiality and security of your application’s sensitive data while seamlessly integrating it into your AKS environment. This approach provides a robust solution for managing secrets, enhancing the overall security posture of your Kubernetes-based applications.

Let’s get started.

Step-1: Create an AKS cluster with Azure Key Vault provider for Secrets Store CSI Driver support.

Note: If you want to use Microsoft Entra Workload ID, you must also use the --enable-oidc-issuer and --enable-workload-identity parameters

az group create -n key-vault -l eastus
az aks create -n AKS-Cluster -g key-vault --enable-addons azure-keyvault-secrets-provider --enable-oidc-issuer --enable-workload-identity

Note: If you have previously created the cluster without the following addons and other parameters, the cluster and addons can still be updated using the commands below.

--addons azure-keyvault-secrets-provider # for Secrets Store CSI Driver support

--enable-oidc-issuer # Used to enable OIDC (OpenID Connect) issuer in Azure AKS.
--enable-workload-identity # Used to enable workload identities in Azure AKS.
--enable-secret-rotation # Used to enable automatic secret rotation in Azure AKS.

• The updating of the cluster and addons with the aforementioned parameters. Remember, if these were not present in your previous cluster, execute these commands.

az aks enable-addons --addons azure-keyvault-secrets-provider --name AKS-Cluster --resource-group key-vault
az aks update --name AKS-Cluster --resource-group key-vault --enable-oidc-issuer --enable-workload-identity --enable-secret-rotation

• Verify the Azure Key Vault provider for Secrets Store CSI Driver installation.

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'

Step-2: Create or use an existing Azure Key Vault: Create a key vault with Azure role-based access control (Azure RBAC). The name of the Key Vault you want to create must be unique.

az keyvault create -n mk-keyvault -g key-vault -l eastus --enable-rbac-authorization

Step-3: Connect your Azure identity provider to the Azure Key Vault Secrets Store CSI Driver in Azure Kubernetes Service (AKS).

Configure workload identity

1- Configure workload identity

export SUBSCRIPTION_ID=ddf05ef9-...................
export RESOURCE_GROUP=key-vault
export UAMI=azurekeyvaultsecretsprovider-aks-cluster
export KEYVAULT_NAME=mk-keyvault
export CLUSTER_NAME=AKS-Cluster

az account set --subscription $SUBSCRIPTION_ID

2- Create a managed identity using the az identity create command.

az identity create --name $UAMI --resource-group $RESOURCE_GROUP

export USER_ASSIGNED_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)"
export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

3- Create a role assignment that grants the workload identity permission to access the key vault secrets, access keys, and certificates using the az role assignment create command.

export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)

az role assignment create --role "Key Vault Administrator" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

• Let’s check role assignment..

4- Get the AKS cluster OIDC Issuer URL using the az aks show command.

export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo $AKS_OIDC_ISSUER

5- Get the object ID of the Microsoft Entra application using the following commands. Make sure to update the values for serviceAccountName and serviceAccountNamespace with the Kubernetes service account name and its namespace.

export SERVICE_ACCOUNT_NAME="workload-identity-sa"  # sample name; can be changed
export SERVICE_ACCOUNT_NAMESPACE="default" # can be changed to namespace of your workload

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
EOF

6- Create the federated identity credential between the managed identity, service account issuer, and subject using the az identity federated-credential create command.

export FEDERATED_IDENTITY_NAME="aksfederatedidentity" # can be changed as needed

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

Important Note: You need to assign users the “Key Vault Administrator” role for the Key Vault you’ve created under the same subscription. Otherwise, you won’t have the permission to create “key”, “secret”, and “certificates” objects.

• Before assigning the role to the user,

• After the role has been assigned to the user,

• Before creating SecretProviderClass and pods, let’s create secrets for a sample database from the portal.

Step-7: Deploy a SecretProviderClass using the kubectl apply command and the following YAML script.

cat <<EOF | kubectl apply -f -
# This is a SecretProviderClass example using workload identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi # needs to be unique per namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "${USER_ASSIGNED_CLIENT_ID}" # Setting this to use workload identity
    keyvaultName: ${KEYVAULT_NAME}       # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: username            # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: password               # Set to the name of your key
          objectType: secret
          objectVersion: ""
    tenantId: "${IDENTITY_TENANT}"        # The tenant ID of the key vault
EOF

Step-8: Deploy a sample pod using the kubectl apply command and the following YAML script.

cat <<EOF | kubectl apply -f -
# This is a sample pod definition for using SecretProviderClass and workload identity to access your key vault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-wi
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-wi"
EOF

• Validate Key Vault secrets.

kubectl exec busybox-secrets-store-inline-wi -- ls /mnt/secrets-store/
kubectl exec busybox-secrets-store-inline-wi -- cat /mnt/secrets-store/username
kubectl exec busybox-secrets-store-inline-wi -- cat /mnt/secrets-store/password

How to load secrets in to environment variable on POD

To use these secrets as environment variables inside the pod, we will add the following blocks to the SecretProviderClass and pod YAML files. Then let’s deploy the yaml files.

# secretproviderclass.yaml (secretObjects block.)
...
spec:
  provider: azure
  secretObjects:
  - data:
    - key: username
      objectName: username
    secretName: username
    type: Opaque
  - data:
    - key: password
      objectName: password
    secretName: password
    type: Opaque 
...

pod.yaml (env block)
...
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    ...
      env:
      - name: DB_USER_NAME
        valueFrom:
          secretKeyRef:
            name: username
            key: username
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: password
            key: password
 ...

• With the following commands, we can verify that the secrets are embedded as environmental variables within the pod.

k exec busybox-secrets-store-inline-wi -- env | grep DB_USER_NAME
k exec busybox-secrets-store-inline-wi -- env | grep DB_PASSWORD
k describe po busybox-secrets-store-inline-wi

• Don’t forget to delete all resources.

az group delete -n key-vault
az group delete -n MC_key-vault_AKS-Cluster_eastus

• Thank you for adding my article to your reading list! If you enjoyed it and found it helpful, please consider following me and giving the article a clap. Your support means a lot and helps me continue creating content that you love.
• Thanks again, and happy reading!