Building ArgoCD Ecosystem With Secret Management — The GitOps Way (Part II)
The secret tool research
Once we’ve deployed our ArgoCD servers in IaC way, in order to start using GitOps approach in our Kubernetes products with multiple environments (development, staging and production) it was the right time to handle the Secrets management challenge.
The legacy deployments in our products used the CI tool (Jenkins) with Jenkins credentials in order to push secrets to the helm values files or just manual creation (bad idea, we know that 🙂), so in order to create our Kubernetes charts on-top our new ArgoCD servers, we came to an understanding that we had to change our secrets management.
First step in solving the challenge would be — defining the problem and the requirements:
- Easy integration with ArgoCD
- Support HashiCorp Vault
- Working with all Kubernetes resources (some of our application charts includes secrets inside helpers and config maps)
- Integration with Helm
After a market research we’ve conducted, based on our requirements, we came to a conclusion that our choice is between Argo CD Vault Plugin and External Secrets Operator.
Because of ArgoCD Vault plugin being part of the ArgoCD ecosystem, we’ve chosen to begin our PoC with it.
How the Argo CD Vault Plugin being installed
Using Terraform value templating we’ve demonstrated in part 1, we can add vault plugin built-in integration, first creating a secret with vault authentication credentials with chosen authentication method, you can find the example here.
Adding it to the ArgoCD configuration template file using the following configuration.
How the Argo CD Vault Plugin Works
The Argo CD Vault Plugin works by replacing the pattern of <placeholder> in any kubernetes YAML or JSON manifests with the secrets values taken from Vault.
An annotation can be used to specify exactly the path that the plugin should look the Vault values. The annotation needs to be in the following format:
kind: Secret
apiVersion: v1
metadata:
name: secrect-with-annotation
annotations:
avp.kubernetes.io/path: "path/to/password"
type: Opaque
stringData:
password: <password>Another way to specify the Vault secret path is using inline-path placeholders. This way allows us to specify path and key for multiple distinct secrets into the same YAML:
kind: Secret
apiVersion: v1
metadata:
name: secrect-without-annotation
type: Opaque
stringData:
username: <path:some/path/to/usernames#secret-key>
password: <path:some/path/to/passwords#secret-key>Once the plugin has finished replacing the placeholders with the secrets, ArgoCD applies the updated manifest to the Kubernetes cluster.
Prerequisites
In this tutorial we wont cover how to install Vault server, but you can find here the Vault installation documentation.
Deploy an ArgoCD application to development environment using the ArgoCD Vault Plugin with helm values file
First thing, we need to enable a Vault kv-v2 store:
$ vault secrets enable -version=2 argocdThen, we need to create the kv secrets in vault for development environment.
Development secret:
$ vault kv put argocd/development username=my_dev_user password=my_dev_pass===== Secret Path =====
argocd/data/development======= Metadata =======
Key Value
--- -----
created_time 2022-10-31T20:46:20.138968925Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
Global secret:
$ vault kv put argocd/global globalToken=Mytoken=== Secret Path ===
argocd/data/global======= Metadata =======
Key Value
--- -----
created_time 2022-10-31T21:03:07.745292476Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
Now, let’s go back to our example of the helm chart application, which containing generic nginx with two templated secret manifests, env_secret.yaml and inline_path_secret.yaml. You can find this chart example on GitHub repository under the helm directory.
It’s worth paying attention to the templates secrets manifests, that represent two different ways of plugin usage.
annotation usage (env_secret.yaml):
apiVersion: v1
kind: Secret
metadata:
name: my-secret-example
annotations:
avp.kubernetes.io/path: "argocd/data/{{ .Values.env }}"
type: Opaque
stringData:
password: <password>inline-path usage (inline_path_secret.yaml):
kind: Secret
apiVersion: v1
metadata:
name: secrect-without-annotation
type: Opaque
stringData:
username: <path:argocd/data/{{ .Values.env }}#username>
token: <path:argocd/data/global#globalToken>Finally, we need to create the ArgoCD application for development environment.
Can be saved as argo_app_dev.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argo-dev-app
namespace: argocd # The namespace must match the namespace of your Argo CD instance - typically this is argocd.finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: default # application namespace
server: https://kubernetes.default.svc
project: default # argo project
source:
path: helm # Path to appsPath with the template name
repoURL: git@github.com:Transmit-Security-Engineering/argocd-with-secrets-mgmt.git # our repo with the chart example
targetRevision: main
plugin:
name: argocd-vault-plugin-helm-with-args
env:
- name: HELM_ARGS
value: -f dev.values.yaml # here we configured the values file of our environment
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
---
Create the ArgoCD application with kubectl:
$ kubectl create -f argo_app_dev.yamlTime to check if we can see the application in the ArgoCD UI:
We can validate our secrets values using the following commands:
$ kubectl get secret my-secret-example -o jsonpath="{.data.password}" | base64 --decode
my_dev_pass%$ kubectl get secret secrect-without-annotation -o jsonpath="{.data.token}" | base64 --decode
Mytoken%$ kubectl get secret secrect-without-annotation -o jsonpath="{.data.username}" | base64 --decode
my_dev_user%
Changing the Vault Secret
What about the support of changing/rotating the secrets?
Time to figure out how it works!
First, update our development password secret in vault:
vault kv put argocd/development password=my_new_dev_pass===== Secret Path =====
argocd/data/development======= Metadata =======
Key Value
--- -----
created_time 2022-11-01T00:20:08.127676478Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 2
Performing a manual hard refresh to the application via the ArgoCD UI, will make a dry run of the plugin:
Then, the ArgoCD Vault Plugin will update the secret from Vault. ArgoCD will perform a sync to the application and will update the Kubernetes secret.
And we’re good to go:
$ kubectl get secret my-secret-example -o jsonpath="{.data.password}" | base64 --decode
my_new_dev_pass%So it’s time to make a decision.
In the beginning, the implementation of the ArgoCD Vault Plugin works great for us. The plugin support with all Kubernetes resources was the best feet for our helm application structure. In addition, the plugin did not require an Operator or Custom Resource to manage and maintain.
But then, we ran into some issues…
The problem started when some of our developers teams requested an auto sync secrets support.
Unfortunately, sync secrets in the plugin working in manual way or can be implement with custom webhook/CronJob (auto refresh of secrets probably won’t be included in the next Argo CD Vault Plugin versions —reference to the GitHub issue).
The sync disadvantage and the fact that the plugin can be configured to work with only one secret manager at time, brought us to start testing the second option, the External Secrets Operator.
In this post, you have learned how to deploy an helm chart with values file on ArgoCD server, taking the secrets from Vault and using the Argo CD Vault Plugin as our secret management tool.
On our next blog, we would like to focus on External Secrets Operator with ArgoCD integration — so make sure you read the third part!
