Enabling GitOps in k8s with ArgoCD with SOPS and AWS KMS support

After reading the great article that Frank wrote about enabling GitOps in k8s, we decided to take some time on our labs testing the most popular GitOps tool out there. This article will give you an overview about ArgoCD.

What ArgoCD is?

Take a look at the Argo-CD webpage to get an idea on what ArgoCD is.

Secret Management on GitOps

ArgoCD remains unopinionated about how to handle secrets, on our projects we like to use SOPS, storing bits isn’t as expensive as processing them, solutions like sops can use some of your existent Key Management platform (as AWS KMS, GCP GMS, Azure Key Vault and PGP), and because of the nature of the tool, the encrypted files could be stores on a git repository.

The first step, is to create KMS key, just follow the AWS’s documentation to do it.

After creating the key, Save the arn, should be something like: arn:aws:kms:us-east-2:270179619257:key/d8bf4685-590e-49b6-8c05-abfabff7aa96

Assuming that you had your AWS’s credentials configured on a ~/.aws/credentials, export the ARN as a variable, and crypt a values.yaml file with sops, in general we use a namespace to hold an environment, so our files are something like enc.${NAMESPACE}.values.yaml, where NAMESPACE will be the name of a defined namespace in the cluster, for example: 3xmlab.

export SOPS_KMS_ARN="arn:aws:kms:us-east-2:270179619257:key/d8bf4685-590e-49b6-8c05-abfabff7aa96"
sops -e values.yaml > enc.3xmlab.values.yaml

You could also do it exporting your PGP’s fingerprint into a SOPS_PGP_FP variable.

export SOPS_PGP_FP="049f0d6f5bdd4670227ff70da87efd8ebc12bf0b"

Or create a rules files .sops.yaml

creation_rules:
        - pgp: '049F0D6F5BDD4670227FF70DA87EFD8EBC12BF0B'
          kms: 'arn:aws:kms:us-east-2:270179619257:key/d8bf4685-590e-49b6-8c05-abfabff7aa96'

Custom argocd-repo-server image

ArgoCD has two approaches to add additional tools to our gitops universe, one is to add the binaries via volume mount, the other one, is to use a custom image.

In our case, we had chosen the second approach, you can find the Dockerfile here, the image is regularly built by our Ñamandú tool and published into docker hub.

Install argocd and create a Plugin

We had modified a .yaml file according to our needs, that is, changing the image to use, and adding a section to a configmap that argoCD use:

apiVersion: v1
kind: ConfigMap
data:
  configManagementPlugins: |
    - name: sopsNdu
      init:
        command: ["/bin/sh", "-c"]
        args: [ "helm init --client-only && helm dependency build" ]
      generate:   
        command: ["/bin/sh", "-c"]
        args: [ "sops -d enc.${ARGOCD_APP_NAMESPACE}.values.yaml > ${ARGOCD_APP_NAMESPACE}.values.yaml && helm template -f ${ARGOCD_APP_NAMESPACE}.values.yaml . "  ]
  repositories: |
    - type: git
      url: https://gitlab.com/iotops/helmcharts/gitops.git
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm

We defined the Plugin, with two sections, the init and the generate.

The init section is optional, and can be used to initialize the directory, in our case we will init helm to work only in client mode, and build any dependency (no tillers will be harmed on the creation of this plugin).

The second section, should generate a valid .yaml output, argoCD will take that output and render it. What we did was to take a encrypted file with the NAMESPACE into the name of it, decrypt it, and then use it as a value file.

So, to install argocd with our custom plugin, run:

kubectl apply -n argocd -f https://gitlab.com/iotops/helmcharts/gitops/-/raw/master/manifests/argocd/install.yaml

Check until the pods goes to Running state.

$ kubectl get pods -n argocd
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-redis-8c568b5db-mvdmb                     1/1     Running   0          47h
argocd-dex-server-76cbdf65d4-f55bk               1/1     Running   0          47h
argocd-application-controller-6c869db745-ss8vh   1/1     Running   0          47h
argocd-server-b54756f69-z9nj5                    1/1     Running   0          47h
argocd-repo-server-7bcc644547-4vc9r              1/1     Running   0          47h

An important detail, take note of the name of the argocd-server pod, if it’s your first install of argocd, that name will be used as the password of the admin’s user, check this link if you want to change it.

Deploy the example application

We had prepare and example application, download the .yaml file and edit it with AWS’s credentials assigned to a role for KMS.

curl -o ~/application.yaml https://gitlab.com/iotops/helmcharts/gitops/-/raw/master/manifests/argocd/application.yaml

After edit the file ~/application.yaml, create a namespace for your application (we use 3xmlab), and run a kubectl apply -f ~/application.yaml, this will deploy an application on your cluster!

kubectl create ns 3xmlab
kubectl apply -f ~/application.yaml

Check the results

If everything went right, you will be able to check your example application deployed

$ kubectl get pods -n 3xmlab
NAME                                        READY   STATUS    RESTARTS   AGE
release-name-hello-world-5d444ccdf7-tj474   1/1     Running   0          47h

Also, you could create a port-forward to the ui of ArgoCD, access it, and see the deployed application.

kubectl port-forward -n argocd service/argocd-server 8080:443

Clap this article if you liked it, and let us know any thought posting a comment.