How To Manage Secrets with GitOps

There are multiple ways to manage secrets with gitops, the most common are:

1. Commit encrypt secrets in git
2. Commit reference to secrets in git

In this blog I will elaborate on two tools from the second option, commit reference to secrets in git that I tried to use: secret store csi driver and external secrets.

I use ArgoCD as the gitops controller but you can use other gitops implementations.

Secrets Store CSI Driver

I started with secrets store CSI driver because it has a sig group.

Secrets store CSI driver works with two daemonsets: one for the csi driver and one for the secret provider. I store my secrets on AWS secret manager so I use AWS provider.

This is how the secret store csi driver works with AWS secret manager:

Secret Store CSI driver

1. Create secret in AWS secret manager
2. Create a secret provider class (reference to the secret in AWS secret manager)
3. Create IRSA — Create IAM role that has access to AWS secret manager from kubernetes cluster (EKS).
4. Add serviceaccount to your deployment that use the created IRSA.
5. Add new volume to your deployment (or other workload) that use the secret provider class and add a mount to this volume
6. If you want to use the secrets in variables you must sync it to kubernetes secrets
7. Only after the pod start and mount the secret you can see the kubernetes secret like any other secret. when last pod that consume the secret is deleted the kubernetes secret is also deleted.

What I didn’t like in secret store csi driver implementation:

1. Two daemonsets — It use two daemonsets and it was too much resources for my use case which was sync secrets from AWS secret manager to kubernetes secrets in order to use them in environment variables.
2. Add serviceaccount to our deployment — You need to add serviceaccount to each deployment or other workload that has access to AWS secret manager.
3. Add volume to a deployment — In order to use the secrets you need to change your deployment in order to add a volume and a mount point that mount the secret.
4. Both 2 and 3 are even more ennoying when you want to add secrets to third party services like argocd and grafana for example.

External Secrets

After working few weeks with Secret store CSI driver I started to look for new solution and saw that external secrets was the most recommended.

External secrets works with operator (deployment) that sync AWS secret manager secrets to kubernetes secret.

These are the steps to work with external secrets operator (ESO):

external secret with aws secret manager

1. Create secret in AWS secret manager
2. Create IRSA — Create IAM role that has access to AWS secret manager from kubernetes cluster (EKS).
3. Create secret store resource — This resource tells ESO how to authenticate to AWS to get the secret and it use the IRSA.
4. Create external secret — this resource tells ESO what to sync from AWS secret manager to kubernetes secret

Summary

For our use case it was much easier to use external secrets because it decouple the secret from the workload (deployment).

External secret allow to configure reference to a secrets and the External Secret Operator sync this secret to a kubernetes secret that you can use like any regular kubernetes secret without making changes in your workload (deployment).