GitOps with Anthos Config Management and multiple sources of truth

This article shows how you can configure AnthosConfigManagement to be used as a tool for deploying apps into your GKE clusters following GitOps methodology with multiple sources of truth.

GitOps is a process of automating infrastructure deployment that is using Git repositories as single source of truth for the infrastructure code. Nowadays, it is considered best practice for organizations managing Kubernetes configuration at scale.

Anthos Config Management aka ACM

ACM being part of Google Cloud offering comes into play when you want to deploy and centrally manage applications and configuration of your GKE clusters. It consist of a few components which all can be used together (but don’t have to)

• Config Sync
• Config Controller
• Policy Controller
• Config connector

Let me briefely introduce those components:

Policy Controller is reponsible for enforcing programmable policies — so called constraints — on the GKE cluster to make sure it is in desired state.

Config Connector — it is used to create Google Cloud resources using Kubernetes Resource Model (KRM) and kubernetes manifests, by making the necessary Google Cloud API calls.

Config Sync — is responsible for synchronisation and reconciliation of clusters to central location set of configuration stored in GIT repositories which then is applied across GKE clusters.

Config Controller — Merge Config Connector with Policy Controller and Config Sync in one component.

Let’s take a closer look into Config Sync component.

In most common scenario, Config Sync is configured with so called RootSync object which points to specific Git repository. It than reconciles and applies if required all resources specified in this repository being single source of truth on GKE cluster.

But we can also use Config Sync for deploying configuration of resources stored in multiple sources of truth. This way we allow deploying ie: multiple applications from multiple repositories and have each repository\app managed by separate team.

Configuring ACM with multiple sources of truth

To achieve this, we can configure ACM with RootSync object which points to repository in which we have defined additional RootSync objects. This additional RootSyncs are pointing to app repositories which hosts code for specific configuration\apps for GKE clusters, like on the diagram below.

Multiple Sources of truth for ACM

1. GKE with ACM configured with RootSync object syncs and reconciles Root_Sync_Repo repository
2. As Root_Sync_Repo repository contains additional RootSync objects pointing to App repositories, ACM configures those app repositories as additional sources of truth (if they are applicable to specific GKE cluster — a bit more about this later)
3. ACM syncs and reconciles content of those app repositories and applies to GKE cluster

Using this approach we can control which app repositories and resources should be synced to which clusters by using objects called ClusterSelectors, Cluster config manifests with specific labels and annotations.

To implement such approach , we need to prepare parent repository which will be configured as main source of truth for ACM on all GKE clusters . Inside the repository we need to:

1. Create Cluster config manifest

Cluster config needs to contain unique name for your GKE cluster and label used by ClusterSelector to limit scope of the RootSync object to only specific GKE clusters based on labels.

The following example define Cluster config with label: application: app_2

kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
 name: cluster-1
 labels:
   application: app_2

NOTE: Labels in Cluster config manifests are not the same as GKE cluster labels set directly on GKE cluster using GUI console, gcloud command or terraform (nor they are visible in console or via gcloud).

2) Create ClusterSelector manifest

Example below configures ClusterSelector which match every GKE cluster defined in Cluster config which has label application: app_2 set (like in Cluster config manifest above)

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: selector-app2
spec:
  selector:
    matchLabels:
      application: app_2

3) Create RootSync object

Example below configures RootSync object for repo https://github.com/arb/acm-sync-app_1.git with annotation: configmanagement.gke.io/cluster-selector: selector-app2

This means that RootSync object app_1 will be scoped only on clusters which match labels set in ClusterSelector named selector-app2, so in our case that would be cluster-1

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: rootsync_app1
  namespace: app_1
  annotations:
    configmanagement.gke.io/cluster-selector: selector-app2
spec:
  sourceType: git
  sourceFormat: unstructured
  git:
    repo: https://github.com/arb/acm-sync-app_1.git
    branch: main
    dir: "/env/dev/"
    auth: token
    secretRef:
      name: git-creds
    noSSLVerify: true

If we want to make RootSync object as mandatory deployment to all of our clusters, we should not specify ClusterSelector annotation in the RootSync manifest. All resources stored in Root repository which doesn’t contain annotation will be deployed to all GKE clusters configured with this parent RootSync as source of truth.

Root repository structure

An example of root repository which is configured as main source of truth can be like below, where specific folders are:

cluster_selectors — this folder contains cluster_selectors definition

clusters folder — contains cluster config manifests

rootsync_objects — contains RootSync for all child repositories that will be synced and applied with ACM

.
├── cluster_selectors
│   ├── clusterselector_app2.yaml
│   └── clusterselector_app3.yaml
├── clusters
│   ├── cluster1.yaml
│   ├── cluster2.yaml
│   └── cluster3.yaml
└── rootsync_objects
    ├── app1
    │   └── rootsync1.yaml
    ├── app2
    │   └── rootsync2.yaml
    └── app3
        └── rootsync3.yaml

With such implementation ACM will deploy resources or config from repositories specified in RootSync objects to clusters based on labels specified in cluster config manifests

Of course ACM is much bigger service than what I just showed and you can configure it many other ways, using not only RootSync objects but also RepoSync or object manigests (like RoleBinding). You can also used different repo structure or even OCI instead of Git repository. But this would be story for other occasion.