ArgoCD + Gitlab + GitOps + Okta + Kustomize in 2024

Published in

Geek Culture

7 min readOct 11, 2022

Today I’m going to release the first article of my work behind ArgoCD and how to implement a stable GitOps pattern using GitLab.

Episode 1: ArgoCD + Gitlab + GitOps + Okta + Kustomize in 2024
Episode 2: ArgoCD root source code and notifications debugging (coming soon, get a ping!)
Episode 3: Gitlab Integration (coming soon, get a ping!)
Episode 4: Final considerations and security concern (coming soon, get a ping!)

The goal is to deploy ArgoCD and define a pattern to deploy applications across different projects using different Gitlab repositories.

Disclaimer:
I saw multiple articles that are a copy-paste from the ArgoCD documentation and that say “GitOps”. Good luck, the 95% are using the wrong git commands and probably they never used it after the demo. For that reason, I’m writing this blog and is up to you to re-shaping it. I’m more than happy to receive some feedback in the comments!

Features:

ArgoCD version 2.4.11
SAML authentication using Okta.
ArgoCD authorization using RBAC.
ArgoCD orphan resources enabled
ArgoCD notification with slack (and how to debug/write them).
Gitlab CI for child pipeline and Gitops pattern.
Custom ArgoCD image with specific kustomize and helm versions.
Kustomize v4.5.7 with a lot of kustomization.yaml!

To understand the articles of this series you need some prior knowledge on:

Kustomize: we deploy ArgoCD with it.
ArgoCD: continuous deployment tool.
Helm: used to deploy a sample application.
GitlabCI: manage the CI for the GitOps pattern.
GitOps: use git as the only source of truth.

If you have specific questions, ask in the comments because I’m not going to explain everything and every dot and comma in the implementation otherwise it requires ages!

Any clap, follow, or comment is highly appreciated!

Let’s plot what we would like to have:

ArgoCD is installed in every environment/cluster.
Some overlay in the argocd-root repository to manage the different environments.

The deployment of ArgoCD is a constellation of IF statements where you have to choose between different conditions.

My first decision has been to deploy ArgoCD using Kustomize for two main reasons:

It’s the officially supported way (The Helm chart is currently community maintained).
I’m comfortable with Kustomize, I think that is a great tool for some use cases. There are thousands of tutorials to use it but is easier to do errors with respect to Helm.

The second decision, HA or not HA?

I follow the High Availability install. If you are here, probably you should follow the same pattern!

Another complex part has been the structure of the repository argocd-root where at root level there are 3 folders:

_backup: Here I put all the manifests for every different release that I’m going to install during the time. I use them for comparison reasons between different versions.
kustomize: contains all the Kustomize overlay of the ArgoCD install.
utils: at the moment contains a script used to generate the admin password for ArgoCD that is using bcrypt.

Using Kustomize as templating mechanism, I defined the structure below in the kustomize folder:

_vendor: this folder contains the plain manifest copy-pasted from a specific version.
base: contains patches and resources that are common in all environments. RBAC policies, notifications, ingress, and root-repository.
dev/stg/prod: environment-specific patches. Admin password, SAML configuration, notifications overlay.

In this kustomizefolder I define the characteristics that my installation MUST satisfy across all the environments. Here I’m managing all the configurations of ArgoCD.

The challenge in my setup is to use only patches to obtain what I was expecting. Multiple times I thought, let’s update something in the _vendor folder to simplify my life. But I prefer the Hard Way also if it requires more time because is easier to maintain.

The final structure looks like this:

.
├── _backup
│   ├── ha_install_2_4_11.yaml
│   └── ha_install_2_4_8.yaml
├── kustomize
│   ├── _vendor
│   │   ├── ha_install.yaml
│   │   ├── ha_install_crd.yaml
│   │   └── kustomization.yaml
│   ├── base
│   │   ├── argocd-notifications-cm.yaml
│   │   ├── argocd-root-app.yaml
│   │   ├── env
│   │   │   ├── argocd-cmd-params-cm.env
│   │   │   ├── argocd-notifications-secret.env
│   │   │   ├── argocd-rbac-cm-policy.env
│   │   │   └── argocd-rbac-cm.env
│   │   ├── ingress.yaml
│   │   ├── kustomization.yaml
│   │   ├── ns.yaml
│   │   └── root-repository.yaml
│   ├── prod
│   ├── stg
│   └── dev
│       ├── env
│       │   ├── argocd-cm-dex.env
│       │   ├── argocd-cm.env
│       │   ├── argocd-notifications-cm.env
│       │   └── argocd-secret.env
│       └── kustomization.yaml
└── utils
    └── bcrypt_generator.py

Below are some highlights of the base/kustomization.yaml:

base/kustomization.yaml:apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomizationnamespace: argocd
bases:
  - ../_vendorresources:
  - ns.yaml
  - ingress.yaml
  - argocd-root-app.yaml
  - root-repository.yamlpatches:
  - argocd-notifications-cm.yamlsecretGenerator:
- behavior: merge
  envs:
  - env/argocd-notifications-secret.env
  name: argocd-notifications-secretconfigMapGenerator:
- name: argocd-cmd-params-cm
  behavior: merge
  envs:
  - env/argocd-cmd-params-cm.env
- name: argocd-rbac-cm
  behavior: merge
  files:
  - policy.csv=env/argocd-rbac-cm-policy.env
  envs:
  - env/argocd-rbac-cm.env

To accomplish the ArgoCD syntax, for some configMapGenerator there is a mix between files and envs. I’m also using argocd-cmd-params-cm.env to define the environment variable server.insecure=true because in my install I’m using another mechanism to manage authentication and is required to use Dex without having issues with the self-signed certificate.

For a specific environment:

env/kustomization.yaml:apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomizationnamespace: argocdresources:
- ../baseconfigMapGenerator:
- behavior: merge
  files:
  - dex.config=env/argocd-cm-dex.env
  envs:
  - env/argocd-cm.env
  name: argocd-cm
- behavior: merge
  files:
  - context=env/argocd-notifications-cm.env
  name: argocd-notifications-cmsecretGenerator:
- behavior: merge
  envs:
  - env/argocd-secret.env
  name: argocd-secretpatches:
- patch: |-
    - op: replace
      path: /spec/rules/0/host
      value: xxx.xxx.com
  target:
    kind: Ingress
    name: argocd-ingress
- patch: |-
    - op: replace
      path: /spec/source/path
      value: argocd/argocd-root/dev
  target:
    kind: Application
    name: argocd-root-appimages:
- name: quay.io/argoproj/argocd
  # Override and use a specific version of kustomize
  newName: "quay.io/pie-r/argocd"
  newTag: "argocd-2.4.11_helm-3.9.4_other-0.1"

Insights:

argocd-cm: contain the path to a specific kustomize version and the url used by SAML authentication. In this ConfigMap there are also the Dex settings used by Okta.
argocd-notifications-cm: we specify the argocdUrl used in the context of the notification to have a nicer slack notification.
argocd-secret: we defined the admin password.
patches: ingress hostname and argocd-root-app overlay.
images: I want to use a specific version of helm and kustomize, without using the built-in provided by ArgoCD. For this reason, I have also a Dockerfile used to build the ArgoCD image.

What I want to achieve:

In the picture above we can see the three overlays, and the more interesting is the base where we define the argocd-origin-app and the argocd-root repository.

Is important to note that argocd-origin-app is part of the ArgoProject:default. This is because by default ArgoCD creates a project called default and you can fine-tune it from the beginning.

The goal of argocd-origin-app is crucial. Is an ArgoApplication that automatically maintains in sync all the customization that we would like to apply for our current installation like:

ArgoProjects: how many projects do we want to use?
ArgoCD repositories: how many repos are going to use ArgoCD?
Argo App of Apps.

To understand better, I use theargocd-origin-app to define the following resources:

A Project for every group of applications (e.g. project-1, project-2).
The applications manifest (e.g. haproxy, redis..) are in a single repository (repo-1-gitops), and I define an App of Apps (project-1-apps) to force ArgoCD to sync the manifest from this external repository.

Final remarks:

ArgoCD is easy to use and install for the first time and for some PoC. Install ArgoCD for professional use requires a deep understanding of it and some security observation during the installation that you must be aware of before starting. More details in Episode 4.

Are you a Terraform fun?
I’ve written something nice for IaaC too:
https://medium.com/geekculture/from-terralith-to-terraservice-with-terraform-acf990e65578

Follow Me and Subscribe to get the updates on this and the next series!

References:

ArgoCD + Gitlab + GitOps + Okta + Kustomize in 2024

Final remarks:

Written by Pier