Partial Helm values encryption using AWS KMS with ArgoCD
How to encrypt only specific yaml fields in values.yaml, and how to configure ArgoCD to decrypt theses secrets before installing a chart.
When using Helm chart that you don’t manage, you might need to hardcode secrets in the chart’s values.
In this blog post I will explain how to encrypt only specific yaml fields in values.yaml, and how to configure ArgoCD to decrypt theses secrets on-the-fly before installing an Helm release. I will use the Grafana Helm Chart as example, but this can apply to any Helm chart.
Context
When deploying an Helm chart, you might want to insert secret data such as API tokens, username/password or any sensitive data. There are many ways to retrieve credentials from your application without writing them to the value files, but some pre-made charts does not allow for retrieving credentials at runtime out of the box. In our case, Grafana requires plaintext datasources configuration, so secrets are hard-coded in datasources.yaml.
This leads to hazardous practices :
The goal of the solution shown in this article is to provide a way to deploy the chart with the secrets configured without storing them in plaintext in the repository, and without losing the GitOps benefits offered by ArgoCD.
Setup for local encryption with Mozilla SOPS
This section describes how to encrypt and decrypt locally the Helm values you want to protect.
Prerequisites :
- Helm
- Helm secrets plugin
- SOPS
- An AWS KMS CMK programatically accessible in your local environment
- An Helm chart with values you want to encrypt
First, you will need to create a .sops.yaml file to configure SOPS encryption mechanism. This file can be located in same directory your values are located in, or in any parent directory, as the SOPS cli will scan recursively upwards.
You can specify which fields to encrypt, which encryption mechanism to use, and other parameters. An extensive list of parameters can be found in the source code as no similar documentation exists as of today.
From the sops.yaml example file above, the single creation rule has 2 fields specified :
- The ARN of the KMS key that will be used to encrypt/decrypt the data
- A regex that selects which YAML fields to encrypt — In our case “secureJsonData” — All children fields will be encrypted recursively
As a best practice I would recommend writing the least amount possible of plain text in the file that will be partially encrypted, as it is not possible with SOPS to edit a non-encrypted field without decrypting/encrypting the whole file. You can write the encrypted data to secrets.yaml aside values.yaml for example.
Here is an unencrypted secrets.yaml
If you have correctly configured your AWS credentials, you can now run Helm Secrets to encrypt your file using helm secrets decrypt secrets.yaml
Configure ArgoCD to decrypt your files on the fly
Now that we can encrypt/decrypt values using AWS KMS, we must configure ArgoCD to use the same decryption mechanism, else it will apply values in their encrypted form.
There are a few steps to follow :
- Install required tools and make them available to ArgoCD
- Edit the ArgoCD ConfigMap to allow usage of Helm Secrets plugin
- Edit your ArgoCD Application to select witch value file to decrypt
To install the tools, you can either use an init container or create your own ArgoCD image.
Add theses fields to your Helm values for deploying ArgoCD.
This configuration will :
- Create an init container that will download Helm Secrets plugin, SOPS and kubectl in a directory
- Mount this directory to a shared volume between the ArgoCD repo server and the init container
- Set environment variables to configure Helm to use installed tools from the shared volume
Now we can edit the ArgoCD ConfigMap (via the Helm values) to allow usage of Helm Secrets plugin by allowing the usage of the secrets scheme.
Multiple schemes can specified by using a coma separator.
Last step ! We can now edit our ArgoCD application manifest to use secrets.yaml alongside values.yaml , and indicate that decryption is necessary for secrets.yaml .
Note the values.yaml and the secrets.yaml in valueFiles : ArgoCD will use values.yaml, alongside decrypting secrets.yaml and use it in the values. secrets:// has been specified to use the Helm Secrets plugin.
Here it is ! Your ArgoCD application should now be deployed, and our Grafana is secured 🤩
