Solved: GitOps with ArgoCD and Kustomize for multi-region, multi-account microservices

ArgoCD multi-account architecture

TL;DR:

• One gitops repo that contains all deployment files per env per region
• ArgoCD per kubernetes cluster (per env per region)

Preface

ArgoCD is a declarative, GitOps continuous delivery leading solution for Kubernetes.

Kustomize is a native kubernetes configuration management.

We chose kustomize because it’s a simple configuration management for yaml files which we already used and because it supports ad-hoc changes so we don’t need to plan in advance.

In this blog post I will explain how to deploy applications on kubernetes in multi AWS accounts/regions.

In our use case we have a restriction for the deployment configuration: traffic in each region is isolated (security constraint) so we decided to deploy argocd per account/region and it looks as in the picture above.

Note: I didn’t mention secrets in this blog post on purpose. Secrets management is out of scope of this blog and deserves its own post.

Deployment Repo

We created one gitops deployment repository that includes the deployments of all our regions.

The directory structure looks like that:

.
├── base
│ ├── microservice1
│ └── microservice2
└── staging
└── eu-west-1
├── applicationset
│ └── generator
└── team1
├── microservice1
│ └── overlays
├── microservice2
│ └── overlays
└── global

We have a base directory for each app that includes all defaults of our deployment and then we have a folder for each env/region.

In this example we have staging/eu-west-1 which includes the following:

1. Applicationset — An argocd object which generates multiple applications from a template. In this template we will configure the apps that are in the team’s folder:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: team1 
spec:
 generators:
 — list:
     elements:
     — app: global
     — app: microservice1
     — app: microservice2
 template:
   metadata:
     name: ‘{{app}}’
       annotations:
         notifications.argoproj.io/subscribe.on-sync-succeeded.teams: channelName
         notifications.argoproj.io/subscribe.on-sync-failed.teams: channelName
         notifications.argoproj.io/subscribe.on-health-degraded.teams: channelName
   spec:
     project: default
     source:
       repoURL: https://gitlab.local/team1/k8s-deployment-argocd.git
       targetRevision: HEAD
       path: “stg/eu-west-1/team1/{{app}}" 
     destination:
       server: https://kubernetes.default.svc
       namespace: team1
     syncPolicy:
       automated:
         prune: true

1. team1/global — Is an argocd app that contains our global configmap. We will reference this configmap from all our deployments and use it as a regional variables repository.
2. team1/[service] — Is a kustomize app that includes our base of [service] which includes a deployment, configmap and cronjobs for a specific microservice and overlays folder to make changes that are specific to this region.

Summary

ArgoCD architecture as suggested here using Kustomize, works great for multi-cluster, multi-region, multi-account cloud deployments. I hope you have gained some insights as well.

Best,

Yossi Nachum