Encrypting Helm Secrets

5 min readAug 21, 2019

Secrets management has always been a weakness for Helm. Further investigation provided us with couple of solutions and one of the possible option that surfaced was Helm Secrets. This is a plugin that we can install on top of Helm that helps to solve the above use case.

I would not be narrating too much about the Helm secret plugin as there are couple good articles that have discussed the benefits and technical aspect of the tool. Instead I will just focus to share my experience of setting up the plugin and an actual use case of using the plugin inside my Helm chart.

The version of my Helm

Client: &version.Version{SemVer:”v2.14.3", GitCommit:”0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:”clean”}Server: &version.Version{SemVer:”v2.14.3", GitCommit:”0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:”clean”}

High overview to setup the plugin

The plugin uses SOPS for encrypting our yaml file.
The plugin supports AWS KMS, GCP KMS, Azure Key Vault and PGP.
Install the Helm-secret plugin.

$ helm plugin install https://github.com/futuresimple/helm-secrets

4. The article uses PGP to perform the setup. Version of my PGP.

gpg (GnuPG) 2.2.17
libgcrypt 1.8.4

5. Perform the following steps.

Generate our key pair

$ gpg --generate-key

Retrieve the fingerprint for the key pair as highlighted.

$ gpg —-fingerprint

The output should be similar as below

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2021-08-19
/Users/devopscontinens.admin/.gnupg/pubring.kbx
--------------------------------------
pub   rsa2048 2019-08-20 [SC] [expires: 2021-08-19]
9B08 DC57 18C3 8BA1 160D  EE53 4115 C1D9 D94B D9B2
uid           [ultimate] devopscontinens <admin@devopscontinens.com>
sub   rsa2048 2019-08-20 [E] [expires: 2021-08-19]

Experience: Avoid using passphase when performing the above step as we will not have the opportunity to manually enter the passphase during decrypting process.

6. Create the directory helm_vars under your Helm chart root directory.

7. Create a file .sops.yaml inside helm_vars folder. Copy our fingerprint value into the following format.

creation_rules:
    -pgp:"9B08 DC57 18C3 8BA1 160D  EE53 4115 C1D9 D94B D9B2"

8. Create a file secrets.yaml inside helm_vars folder. Supply with our key pair value in plain text.

mysecret:pAssw0rd

9. Lets try to encrypt our secrets.yaml using Helm-secret plugin. Helm secret commands starts with “helm secrets <options>”. We can do a helm secrets to have a look at the list of options. Remember to path into the helm_vars directory before invoking the command.

$ helm secrets enc secrets.yaml
Encrypting secrets.yaml
Encrypted secrets.yaml

After executing the above, we will notice the secrets.yaml will be encrypted to a format similar as below.

mysecret:ENC[AES256_GCM,data:sxfEX+kK0U4=,iv:55BozyMoAIB8dD7i3JcOtlzQO6gkwVfSx90J+27y/SY=,tag:mCHsxJVdFxZ6h9mfkcAG8A==,type:str]
sops:
kms:[]
gcp_kms:[]
azure_kv:[]
lastmodified:'2019-08-20T13:40:56Z'
mac:ENC[AES256_GCM,data:28KV+jAT+L7lZSTiIJTL7XC5XvPH4Vzc3R/P/KOdUwBoLBt8Ozo9Z6qQMn4QI6XBYhS127GhD9xOLwMnzjm1yEXxM1dRUpy68jzczDghmUXJx494ZK4klIGEDoQLMaGI6s4rAQoaflix8Tewo3H0ZmQH4P3H/oxcPhRURJY2qns=,iv:f6ZY1L5/Dg9zcIwO5CO8RZ3weQXsHa4+ufkf/iM3GUo=,tag:ivGMwTGyG27JnbQLzsTlXA==,type:str]
pgp:
-created_at:'2019-08-20T13:40:54Z'
enc:|
-----BEGINPGPMESSAGE-----
hQEMA94e2vEPuuuwAQf/fj/fbhRG7w9OeqAmyMKu6UQzEA7HD54287WGbNSmihAc
SgGsUpcgPLRLPO+n2MOrHLlPdx15gUIFfE7q6y1POcYJmTCHiolNDigQFSQj5mQZ
rZ3xT0kzjcrOw8q0HUoHcKgQsn7jQr3Y3MX3z/63wq9jW2lDCqoYk61s7z5SUKLD
J13Q+TUvCEXVfkjO/n2lmuSEgV7rmc5Gq5GkQ3o07hbcJTmCXkapo/s11I4oRMjw
ssxz1psquvx8awKG/SfVhmVllRCIQTINDhCGUHhAqPHhDQ/v79Akh4Snk6Q2Pxfw
QcWJsJvYCMyU68F28j3SNOKkqtHcDOGNSWMATj8+b9JcAZqFXGBjxHdSFUEUs7GJ
tAOH1uBnzW6Jmr3cMluX1mb3YNSG5VYTJ1fmpK/fEgOCZylz91MYt0mMmh+GNVTS
IBN5Z/7V3HAGL24Sq8/+SNB9OZGecyD29Xhcxlw=
=vSV3
-----ENDPGPMESSAGE-----
fp:1189AF85735D53F1285FBBD59EE16A2EC4F2E8FB
unencrypted_suffix:_unencrypted
version:3.3.1

10. We can try to decrypt the secrets.yaml.

$ helm secrets dec values.yaml
Decrypting values.yaml
Not encrypted: values.yaml

A secrets.yaml.dec will be generated which is our decrypted secrets.yaml file.

Experience: Mac users will need to perform an additional step else the decrypt step will fail.

$ gpg --export-secret-keys >~/.gnupg/secring.gpg

With that we can successfully conclude that our Helm-secret plugin is functioning. Let’s move over to our Helm chart implementation.

11. In our example, we will be using both the secrets and values features of the helm-secret. We have already created our secrets.yaml earlier. Let’s create our values.yaml file inside the helm_vars folder. Supply with our key pair value in plain text. Note that we do not perform encryptions for values.

Our values.yaml file.

myvalue:"unencryptedvalue"

12. Our helm chart directory structure should be similar as follow

13. We will create our Kubernetes configmap resource and secret resource inside the helm chart to utilise the secrets and values. These are standard files that will reside inside the /templates folder.

Note that we are using 2 values here. One is the standard configmap resource data and the other is from the helm-secret values.yaml.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Values.configmap.name }}
data:
  {{ .Values.configmap.databasename_key }}:{{   .Values.configmap.databasename_value }}
  myvaluecm:  {{ .Values.myvalue }}

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: event-secret
  labels:
    app.kubernetes.io/name: {{ include "EventService-chart.name" }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  my_secret_key: {{ .Values.mysecret | b64enc | quote }}

Experience: The value of the key/pair value must match our key defined inside the secrets & values inside the helm_vars folder. For example in our configmap.yaml, the myvalue inside {{ .Value.myvalue }} value must be exact match to the key inside our helms_var/values.yaml key (i.e. myvalue).

14. Lastly we can update our deployment.yaml to utilise these values. Take note of the keys that both configMapKeyRef and secretKeyRef are referencing.

deployment.yaml

containers:
  - name: {{ .Chart.Name }}
    image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
    imagePullPolicy: {{ .Values.image.pullPolicy }}
    ports:
      - name: http
        containerPort: 8090
        protocol: TCP
    env: 
      - name: SPRING_DATASOURCE_URL
        valueFrom:
          configMapKeyRef:
          # The ConfigMap resource we are reading from
          name: {{ .Values.configmap.name }}
          # This key is read from our standard configmap resource
          key: {{ .Values.configmap.databasename_url_key }}
      
      - name: SPRING_DATASOURCE_URL
        valueFrom:
          configMapKeyRef:
          # The ConfigMap resource we are reading from 
          name: {{ .Values.configmap.name }}
          # This key is read from our helm-secret values.yaml
          key: myvaluecm      - name: SPRING_DATASOURCE_PASSWORD
        valueFrom:
          secretKeyRef:
          # The secret resource we are reading from
          name: event-secret
          # This key is read from our helm-secret secrets.yaml
          key: my_secret_key

15. Let’s do a dry run deployment of our chart to confirm the results. For this we will use the command “helm secrets install” together with the -f options to pass in the files.

$ helm secrets install ./EventService-chart/ -f ./EventService-chart/helm_vars/secrets.yaml -f ./EventService-chart/helm_vars/values.yaml — dry-run — debug

Below are the values in run time. The encrypted secrets and configmap values are generated and passed into our deployment chart.

---
# Source: EventService-chart/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: event-secret
  labels:
    app.kubernetes.io/name: EventService-chart
    chart: "EventService-chart-0.1.0"
    release: "volted-quokka"
    heritage: "Tiller"
type: Opaque
data:
  my_secret_key: "cEFzc3cwcmQ="---
# Source: EventService-chart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ms-event-config
data:
  spring.datasource.url: jdbc:mysql://localhost:3306/databaseURL
  myvaluecm:  unencrypted value---containers:
  - name: EventService-chart
    image: "chartmuseum/eventservice:stable"
    imagePullPolicy: IfNotPresent
    ports:
      - name: http
        containerPort: 8090
        protocol: TCP
    env: 
      - name: SPRING_DATASOURCE_URL
        valueFrom:
          configMapKeyRef:
          # The ConfigMap resource we are reading from
          name: ms-event-config
          # This key is read from our standard configmap resource
          key: spring.datasource.url
      
      - name: SPRING_DATASOURCE_URL
        valueFrom:
          configMapKeyRef:
          # The ConfigMap resource we are reading from 
          name: ms-event-config
          # This key is read from our helm-secret values.yaml
          key: myvaluecm      - name: SPRING_DATASOURCE_PASSWORD
        valueFrom:
          secretKeyRef:
          # The secret resource we are reading from
          name: event-secret
          # This key is read from our helm-secret secrets.yaml
          key: my_secret_key

Encrypting Helm Secrets

Written by Melvin