Open in app

Sign in

Write

Sign in

ArgoCD with Helm-Secrets and KMS encryption.

Maciej Przygrodzki
7 min readJan 8, 2023

--

Argo CD is a great CD tool which runs as K8s controller which continuously monitors running applications and compares it to current state in repository. I would like to focus on secrets management as values for Helm charts which can be safely kept in your code repository. There are many other tools which can be used as secrets-manager and allows us to instantly commit our secrets but we will focus today on combination with Mozilla SOPS, AWS KMS CMK, and Helm Secrets plugin and ArgoCD at the top of it.

To begin we need to meet some requirements and have following packages installed, most of the commands will be strictly linked with Mac OSX which is my main OS.

Prerequisites:

  • Terraform v1.3.6
brew install terraform
  • ArgoCD (I’ll skip installation process as there are many docs about it)
  • Mozilla SOPS
brew install sops
  • Helm3 (minimum v3.9) and Helm-Secrets plugin
brew install helm
helm plugin install https://github.com/jkroepke/helm-secrets
  • aws-cli

When all the packages are ready to use we need to create AWS KMS customer managed key which will be used to encrypt and decrypt our secrets. You have to be careful who will have access to manage KMS keys, as removing the keys may lead to serious problems. It is worth to set longer deletion period. Assuming you have permissions to manage KMS, here’s a Terraform snippet to quickly create KMS keys, and assumable IAM role for ArgoCD service account so the service will have ability to use the keys. In my example I have created 3 keys for each dev/stg/prd environment, to show how we can easily manage different keys for many envs.

Terraform AWS KMS Keys

### AWS KMS CMK for Helm-Secrets encryption

resource "aws_kms_key" "helm_secrets_dev" {
  description             = "GitHub Secrets Encryption Key for DEV env"
  deletion_window_in_days = 7
  enable_key_rotation     = true
}

resource "aws_kms_alias" "helm_secrets_dev" {
  name          = "alias/helm-secret/dev"
  target_key_id = aws_kms_key.helm_secrets_dev.key_id
}

resource "aws_kms_key" "helm_secrets_stg" {
  description             = "GitHub Secrets Encryption Key for STG env"
  deletion_window_in_days = 7
  enable_key_rotation     = true
}

resource "aws_kms_alias" "helm_secrets_stg" {
  name          = "alias/helm-secret/stg"
  target_key_id = aws_kms_key.helm_secrets_stg.key_id
}


resource "aws_kms_key" "helm_secrets_prd" {
  description             = "GitHub Secrets Encryption Key for PRD env"
  deletion_window_in_days = 14
  enable_key_rotation     = true
}

resource "aws_kms_alias" "helm_secrets_prd" {
  name          = "alias/helm-secret/prd"
  target_key_id = aws_kms_key.helm_secrets_prd.key_id
}

Terraform AWS IAM Role ( EKS cluster was created with official Terraform module so keep in mind there are some relations for values and variables from there).

module "argocd_repo_server_iam_role" {
  source                        = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                       = "5.10.0"
  create_role                   = true
  role_name                     = "EKS-argocd-repo-server-${var.eks_cluster_name}"
  provider_url                  = trimprefix(module.eks.cluster_oidc_issuer_url, "https://")
  role_policy_arns              = [aws_iam_policy.argocd_server_iam_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:argocd:argocd-repo-server"]
  tags = {
    Terraform = "true"
  }
}

resource "aws_iam_policy" "argocd_server_iam_policy" {
  name        = "EKS-argocd-server-${var.eks_cluster_name}"
  description = "EKS ArgoCD server policy for cluster ${var.eks_cluster_name}"
  policy      = data.aws_iam_policy_document.argocd_repo_server_kms_iam_policy.json
}

data "aws_iam_policy_document" "argocd_repo_server_kms_iam_policy" {
  statement {
    sid       = "ArgoCDKMS"
    effect    = "Allow"
    resources = [
      aws_kms_key.helm_secrets_dev.arn,
      aws_kms_key.helm_secrets_stg.arn,
      aws_kms_key.helm_secrets_prd.arn,
      ]

    actions = [
      "kms:Decrypt*",
      "kms:Encrypt*",
      "kms:GenerateDataKey",
      "kms:ReEncrypt*",
      "kms:DescribeKey",
    ]
  }
}

Mozilla SOPS

When we already have CMKs ready let’s start with Mozilla SOPS configuration. Basically sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

SOPS gives us ability to create a .sops.yaml file configuration for each code repository as encryption mechanism. With this approach we can use different encryptions mechanism for every path we want. This file can be located in root directory or any other path your values are stored — SOPS cli will use recursively mechanism to scan.

SOPS have also ability to specify i.e. which fields to encrypt and many other parameters. For more information please check the source code repo and take a look on examples provided there → https://github.com/mozilla/sops. In this article we will encrypt whole file.

I’ve create following SOPS config file in my root directory of repository, using path_regex mechanism I’ve declared which KMS key I would like to use to encrypt values for each environment. My entire directory looks following:

├── README.md
├── charts
│   ├── backend
│   │   ├── Chart.yaml
│   │   ├── templates
│   │   └── values.yaml
│   ├── frontend
│   │   ├── Chart.yaml
│   │   ├── templates
│   │   └── values.yaml
├── environment
│   ├── secrets
│   │   ├── dev
│   │   │   └── values-enc.yaml
│   │   ├── prd
│   │   │   └── values-enc.yaml
│   │   └── stg
│   │       └── values-enc.yaml
│   └── values
│   │   ├── dev
│   │   │   └── values.yaml
│   │   ├── prd
│   │   │   └── values.yaml
│   │   └── stg
│   │       └── values.yaml
├── .sops.yaml
creation_rules:
  - path_regex: 'environment/secrets/dev/(.*).yaml'
    kms: 'arn:aws:kms:us-west-2:1234567890:key/1d386961-c432-4418-916c-e3c9d2b7b09b'

  - path_regex: 'environment/secrets/stg/(.*).yaml'
    kms: 'arn:aws:kms:us-west-2:1234567890:key/e7a1d606-e145-4083-b464-2edab0be1d8d'

  - path_regex: 'environment/secrets/prd/(.*).yaml'
    kms: 'arn:aws:kms:us-west-2:1234567890:key/b65b94da-91f4-408e-a948-9986ce025c98'

In following pattern we are setting additional values files for our chart from environment/ directory, where in values/ and secrets/ we have declared our values to be passed into deployment. And here’s all the magic which allows us to keep secrets directly in the repository after encryption.

Let’s say in our values we have passwords, api keys etc. — if that would be directly commited to repository it’s completely out of the best practices and violates most of security rules.

database:
  password: "23tr23fdcwejqfnkjebjhdashDHBFJ"

secrets:
  api:
    TWILIO_AUTH_TOKEN: "T8qdDeuCkKjVGsvKgtE4hxzVjpqCdgE4"
    TWILIO_ACCOUNT_SID: "aHdbD655TEHsXhw2BkD8Q76r2J48brQy"
    TWILIO_API_SECRET: "HS4BRz9MGGpUhfzYccj57pptRMCprrdu"
    TWILIO_API_SID: "xSwAxExPKu8TQV3zq2DXkyHa7rPykxGH"

By creating new template of that file and encrypting it all the problems are gone — this can be achieved by running following command:

helm secrets encrypt environment/secrets/dev/values.yaml > environment/secrets/dev/values-enc.yaml

This will produce file where all content was encrypted using our KMS key and can be stored directly in our repository.

database:
    password: ENC[AES256_GCM,data:xjGED9NIJsQ3heKgFgDeyX5168RksC1Q97ZISHlt,iv:nmA9kOwZGZHvwfj87EltfT04HhKk4UDdR6Iicfcrh1s=,tag:eJ6exGifYam90kl6vkhtgg==,type:str]
secrets:
    api:
        TWILIO_AUTH_TOKEN: ENC[AES256_GCM,data:0ToP+uxed0wA7Fs2xqkgUO3l1h+PU9ZuK+Pdefifi78=,iv:m1M22iASi9jyWCdAZmKxNuhGdbHTRv+H0RAALzpdZec=,tag:Ahnv2K4VkdkFLzf3vYGKyQ==,type:str]
        TWILIO_ACCOUNT_SID: ENC[AES256_GCM,data:1VTM1TkE+T7+i7o3hd+5Oon0lzzVQjsOOj6HVgtHA4E=,iv:jJJz0BFNpUaKS63TPkObmRxQdod/irZ82L6zUEo6Wrk=,tag:zjWIcQbYret5PSp971R1WA==,type:str]
        TWILIO_API_SECRET: ENC[AES256_GCM,data:6KlMIU0kelSTDBzFL5mS2imxcK2SBm9OVg9JcG+0kIU=,iv:n+u4+WtmRyZLzvEkREPU2kdjJBtlfuLQDaGu0qqZXz4=,tag:2V3ZgdYgSWis9jFnN+e1VQ==,type:str]
        TWILIO_API_SID: ENC[AES256_GCM,data:WKHXMvLzaSnQBPWG9bYlVfJq3YU0JrrXtOyFSclV3Ac=,iv:5/pyhtoSc2vfbpCvG8Ioh2JmSXG4Czlial1YA3ddc4o=,tag:oaIIcvOfqcQOBeOCdq5NDg==,type:str]
sops:
    kms:
        - arn: arn:aws:kms:us-west-2:1234567890:key/1d386961-c432-4418-916c-e3c9d2b7b09b
          created_at: "2023-01-08T11:55:21Z"
          enc: AQICAHjiTNb7z0VSaLQfI6ZQl7AAzJLzNPkB6K+1BiIOIo9HfwEkD4hOFO4NEWV9Daj/ssT4AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMg0KGkoNZNtBscFmPAgEQgDsJA6LSUt3pZzn/trnj8updCqZKySsPL+5G/nyVdIyRHNXW5feT5egdz20JEIEMk0s4xqt/JdnbYDCVyA==
          aws_profile: ""
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-01-08T11:55:22Z"
    mac: ENC[AES256_GCM,data:A9P5h7rDcNWDDGzNKZZ4ufzQO53j2FbrDm9SN2OHhBygurYp+EjSBQKNbPvcJLzT2rVv6FDdcs3Zn9ptQ9vDAvSb21J2fb53InY2reyaIqxKa19P6cec75pPlBytUCnd6w8LTUTVhLmbhY6NNT0+/x7ggWvxSVeThrviWwV/ZM0=,iv:e9R875SnM2WChaO6FcLG5/n7t9QacVtJGXM99RkwP08=,tag:qo8P3+4w9Ye9mNDTzV6hXw==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.7.3

What if we would like to modify the content and add additional secrets ? Simply run helm secrets edit <path-to-encrypted-file> command and add new secret. Helm Secrets plugin will automatically re-encrypt the file again and all you need to do is just to commit the changes.

helm secrets edit environment/secrets/dev/values-enc.yaml
database:
    password: ENC[AES256_GCM,data:xjGED9NIJsQ3heKgFgDeyX5168RksC1Q97ZISHlt,iv:nmA9kOwZGZHvwfj87EltfT04HhKk4UDdR6Iicfcrh1s=,tag:eJ6exGifYam90kl6vkhtgg==,type:str]
secrets:
    api:
        EXTRA_SECRET: ENC[AES256_GCM,data:rL7ZzkW/yztf+1Lr1gAPAp9c7zKVn2g=,iv:6j1sJRvugZshTexfVNvIpS0MMgUT+69m85Fnn4JzYZo=,tag:zZbc96Bs7db5zPW90OQY3g==,type:str]
        TWILIO_AUTH_TOKEN: ENC[AES256_GCM,data:0ToP+uxed0wA7Fs2xqkgUO3l1h+PU9ZuK+Pdefifi78=,iv:m1M22iASi9jyWCdAZmKxNuhGdbHTRv+H0RAALzpdZec=,tag:Ahnv2K4VkdkFLzf3vYGKyQ==,type:str]
        TWILIO_ACCOUNT_SID: ENC[AES256_GCM,data:1VTM1TkE+T7+i7o3hd+5Oon0lzzVQjsOOj6HVgtHA4E=,iv:jJJz0BFNpUaKS63TPkObmRxQdod/irZ82L6zUEo6Wrk=,tag:zjWIcQbYret5PSp971R1WA==,type:str]
        TWILIO_API_SECRET: ENC[AES256_GCM,data:6KlMIU0kelSTDBzFL5mS2imxcK2SBm9OVg9JcG+0kIU=,iv:n+u4+WtmRyZLzvEkREPU2kdjJBtlfuLQDaGu0qqZXz4=,tag:2V3ZgdYgSWis9jFnN+e1VQ==,type:str]
        TWILIO_API_SID: ENC[AES256_GCM,data:WKHXMvLzaSnQBPWG9bYlVfJq3YU0JrrXtOyFSclV3Ac=,iv:5/pyhtoSc2vfbpCvG8Ioh2JmSXG4Czlial1YA3ddc4o=,tag:oaIIcvOfqcQOBeOCdq5NDg==,type:str]
sops:
    kms:
        - arn: arn:aws:kms:us-west-2:1234567890:key/1d386961-c432-4418-916c-e3c9d2b7b09b
          created_at: "2023-01-08T11:55:21Z"
          enc: AQICAHjiTNb7z0VSaLQfI6ZQl7AAzJLzNPkB6K+1BiIOIo9HfwEkD4hOFO4NEWV9Daj/ssT4AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMg0KGkoNZNtBscFmPAgEQgDsJA6LSUt3pZzn/trnj8updCqZKySsPL+5G/nyVdIyRHNXW5feT5egdz20JEIEMk0s4xqt/JdnbYDCVyA==
          aws_profile: ""
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-01-08T11:59:43Z"
    mac: ENC[AES256_GCM,data:FMiyBixEMTle63P2iF16bn2DM0EDoYf7ETxaGafOz+arTBYy0B5mNkgzptbUOyshnhI6OtvwDIbSYEZI2+z7mlFWNhVCNH0jfYzX5mnA7z8+lhC3p1Y9MaQ+vuEPtObjPa3rssceEP0QBODTPUUTE2r3M+7xONzHXJJaH8VW2SU=,iv:RLIvQHGbeV00uPmoKlQS9s0PSMT9jLNapf+49JmYiE4=,tag:4Etvn5Ysxahh2mpeHN320A==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.7.3

As you can see above file was re-encrypted and new entry is visible under secrets/api section.

Without ArgoCD you can deploy you chart with this cli i.e.

helm secrets upgrade -i backend charts/backend -f environment/values/dev/values.yaml -f environment/secrets/dev/values-enc.yaml --namespace dev

ArgoCD and Helm-Secrets

At the top of the article I’ve mentioned about using ArgoCD with our encrypted secrets. Thats works like a charm but before we start we need to deploy ArgoCD with some additional values for repo-server. Firstly we need to set IAM Role arn for service-account of the repo-server — I’ve deployed ArgoCD via Helm chart, so all the tips will be related to this type of setup.

For main argocd-server we need to enable it in configmap:

server:
  config:
    helm.valuesFileSchemes: >-
      secrets
repoServer:
  serviceAccount:
    annotations: 
      eks.amazonaws.com/role-arn: "arn:aws:iam::1234567890:role/EKS-argocd-repo-server-dev-eks-local"

Also we need to have ability to use helm-secrets plugin, based on official documentation and with Helm chart we can quickly achieve that by running extra init-container and setting additional environment variables which is that fastest way compared to building a custom docker image.

  env:
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/
    - name: HELM_SECRETS_SOPS_PATH
      value: /custom-tools/sops
    - name: HELM_SECRETS_VALS_PATH
      value: /custom-tools/vals
    - name: HELM_SECRETS_KUBECTL_PATH
      value: /custom-tools/kubectl
    - name: HELM_SECRETS_CURL_PATH
      value: /custom-tools/curl
    - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
      value: "true"
    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "true"
    - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
      value: "true"
  volumes:
    - name: custom-tools
      emptyDir: {}
  volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools

  initContainers:
    - name: download-tools
      image: alpine:latest
      command: [sh, -ec]
      env:
        - name: HELM_SECRETS_VERSION
          value: "4.1.1"
        - name: KUBECTL_VERSION
          value: "1.24.6"
        - name: VALS_VERSION
          value: "0.18.0"
        - name: SOPS_VERSION
          value: "3.7.3"
      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

          wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
          wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

          wget -qO- https://github.com/variantdev/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;
          
          # helm secrets wrapper mode installation (optional)
          # RUN printf '#!/usr/bin/env sh\nexec %s secrets "$@"' "${HELM_SECRETS_HELM_PATH}" >"/usr/local/sbin/helm" && chmod +x "/custom-tools/helm"
          
          chmod +x /custom-tools/*
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

With this kind of setup our ArgoCD have the ability to use helm-secrets and permissions to access CMK keys in AWS KMS to decrypt the secrets values.

To declare secrets in our application manifest, ArgoCD have unique way for helm-secrets and it should like following. Absolute, relative and traversal paths can be accessible by setting ENV variables to true in above manifest. At the time of writing ArgoCD still doesn’t have the ability to use external repositories as Helm values.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backend-dev
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    name: 'dev-eks-local'
    namespace: dev
  source:
    repoURL: 'git@github.com:xxxxxxxxxx/app-infra.git'
    path: charts/backend
    targetRevision: HEAD
    helm:
      valueFiles:
        - ../../environment/values/dev/values.yaml
        # Pattern use helm-secrets
        - secrets://../../environment/secrets/dev/values-enc.yaml
      releaseName: backend
      version: v3
  project: apps
  syncPolicy:
    automated:
      selfHeal: true
      prune: false

And that’s all. With this kind of mixed setup our secrets are safe and secured. Hopefully I will post more soon. Enjoy.

Argocd
Helm
Aws Kms
Secrets
Encryption

--

--

Maciej Przygrodzki

Written by Maciej Przygrodzki

19 Followers

DevOps Engineer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams