GitOps and Secret Management with Kubernetes CSI-Secret-Store on Azure

Gitops defines an environment source of truth in a source version control, Git, or SVN repository. (If you are not familiar with the concept of GitOps, here’s a great video to introduce the topic: What Is GitOps And Why Do We Want It?)

Kubernetes is ideal for GitOps because it works off of configuration files called manifests that describe the desired state and configuration of services. Once manifests are applied, Kubernetes will make sure the desired state is eventually reached. This is perfect for GitOps where all configurations are stored under version control.

Problem:

Storing the desired application state in a repository has a lot of advantages, such as:

1. Clarity of the deployed state.

2. Easy understanding of the application changes over time.

3. Easy rollback: “git revert HEAD~1”.

4. Multiple environments — multiple branches.

Secrets are not code. That means, we cannot store them in Git as plain text. There are multiple ways to store secrets encrypted in Git or use 3rd party services. (We will not cover the use case of encrypting and storing secrets in Git in this article.)

The most popular GitOps operators today, ArgoCD and Flux do not solve the secret management problem. They are indifferent about it for two reasons:

1. The Twelve-Factor App defines that the application should store configuration in environment variables. Not all the applications follow that approach; the application may read a configuration file that is supposed to be on the filesystem or call to remote service during startup.
2. We might store secrets encrypted in Git or use a 3rd party service for storing them. Examples are Azure Key-Vault, Google KMS, AWS KMS, and others.

Solution

Both ArgoCD and Flux (as well as other GitOps operators) have addressed that issue in the past. The ArgoCD team stating: “Argo CD is un-opinionated about how secrets are managed. There are many ways to do it and there’s no one-size-fits-all solution”. Flux has also created a similar post to describe the possible solutions.

A list of existing solutions (from ArgoCD):

• Bitnami Sealed Secrets
• Godaddy Kubernetes External Secrets
• External Secrets Operator
• Hashicorp Vault
• Banzai Cloud Bank-Vaults
• Helm Secrets
• Kustomize secret generator plugins
• AWS-secret-operator
• KSOPS

In this article, we will try another solution: CSI-Secret-Store. CSI-Secret-Store is a subproject of Kubernetes SIG-Auth which defines an interface between secret providers and secret users (Pod, Secret). Today we already have 2 providers that implement that interface: Azure and Hashicorp Vault. We will focus on Azure Key Vault which provides a way to securely store secrets, keys, and certificates.

Installation & Configuration:

Prerequisite

• A Kubernetes cluster that is up and running
• Azure account & Azure CLI authenticated in your terminal

Installation

• Install required manifests for CSI-Driver

export NS=csi-test-run
kubectl create ns $NS
kubectl apply -f https://raw.githubusercontent.com/olegsu/blogs/master/Gitops-and-Secret-Management-with-Azure-CSI-Secret-Store/manifests/csi-driver/csi-driver-global.yaml
kubectl create clusterrolebinding secretproviderclasses-rolebinding --clusterrole secretproviderclasses-role --group rbac.authorization.k8s.io --serviceaccount ${NS}:secrets-store-csi-driver
kubectl apply -n $NS -f https://raw.githubusercontent.com/olegsu/blogs/master/Gitops-and-Secret-Management-with-Azure-CSI-Secret-Store/manifests/csi-driver/csi-driver.yaml

• Install Azure CSI Provider

kubectl apply -n $NS -f https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/deployment/provider-azure-installer.yaml

Create Azure Key-Vault

• Find account related details

az account list -o json
export SUBSCRIPTION_ID= # SubscriptionID is .id of the account
export TENANT_ID= # TenantID is .tenantId

• Create a Resource-Group

export RESOURCE_GROUP_NAME=csi-test-drive
az group create --name $RESOURCE_GROUP_NAME --location eastus

• Create Key-Vault

export KV_NAME=csi-kv
az keyvault create --name $KV_NAME --resource-group $RESOURCE_GROUP_NAME

• Add a secret

az keyvault secret set --vault-name $KV_NAME --name secret-name --value "supersecret"
# print the secret
az keyvault secret show --vault-name $KV_NAME --name secret-name --query value

• Create Service Principal

We will be using Service Principal to fetch the secrets. Other methods are available in the CSI-Azure repo.

export SERVICE_PRINCIPAL_NAME=csi-sp-test-drive
az ad sp create-for-rbac --skip-assignment --name $SERVICE_PRINCIPAL_NAME
export SERVICE_PRINCIPAL_PASSWORD= # Export the password from .password

• Get the Azure ClientID

export AZURE_CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId -o tsv)

• Create Secret with the token to Azure

kubectl create secret -n $NS generic secrets-store-creds --from-literal clientid=$AZURE_CLIENT_ID --from-literal clientsecret=$SERVICE_PRINCIPAL_PASSWORD

• Create Azure Role Assignment

Assign Reader Role to the Service Principal for your Key Vault:

az role assignment create --role Reader --assignee $AZURE_CLIENT_ID --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCE_GROUP_NAME/providers/Microsoft.KeyVault/vaults/$KV_NAME

• Create a Security Policy

Allow Service Principal to get Secrets, Keys, and Certificates from Key Vault:

az keyvault set-policy -n $KV_NAME --key-permissions get --spn $AZURE_CLIENT_ID
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn $AZURE_CLIENT_ID
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn $AZURE_CLIENT_ID

• Create SecretProviderClass

Secret Consumption

Now that everything is installed we can start consuming the secrets. CSI-Secret-Store provides two ways to use a secret in your application:

1. Use a secret in a Pod definition by attaching a volume to the Pod with all the required secrets so the application can load them from the filesystem.
2. Sync to Kubernetes Secret object. In this approach, the CSI-Driver will create and update a Secret object with the value from Key Vault so it can be used from pods or anyone who has permission to read that secret. This approach is riskier, especially when the secrets are not encrypted at rest. See example.

Example: Inject into the Pod filesystem.

• List the attached volume

Once a Pod is been created and running we can list the volume:

kubectl -n $NS exec -it nginx-secrets-store-inline cat /mnt/secrets-store/secret-name

Discussion

Although GitOps is a new term for an old concept, secret management was always a challenge for any organization. In the end, we need to be able to use the secret in the application without compromising the security.

Using CSI-Secret-Store is another approach that helps to use secrets in the application, without storing them in Git. Although the project is still in its very early stage, its development is moving fast.

At this time, we are still missing official providers for AWS and Google.