Build a Modern Platform with Crossplane, Anthos and ArgoCD: The Future of Infrastructure Management

In a world of ever-growing containerized applications and their dependencies, managing cloud infrastructures is becoming increasingly complex.

This article explores how to take a GitOps approach with Crossplane to simplify the deployment and maintenance of modern cloud platforms. By leveraging the principles of Kubernetes and focusing on traceability, collaboration and configuration drift detection, Crossplane offers an attractive alternative to traditional solutions such as Terraform.

In addition, application portability is facilitated by Crossplane’s ability to define abstractions for infrastructure resources, simplifying migration and adoption across multiple clouds. In this context, Anthos is proving to be a valuable asset. As Google’s hybrid and multi-cloud Kubernetes cluster management platform, Anthos facilitates unified management of cloud and on-premises environments, improves application portability, and optimizes development and operations processes.

We will present an architecture that integrates Anthos, Crossplane and ArgoCD to orchestrate infrastructure, ensure configuration consistency and unify application and infrastructure lifecycle management. As a result, teams working with Kubernetes will be able to enjoy a smoother experience, while facilitating the migration and adoption of multiple cloud environments.

Building a modern platform: The 3 must-know tools: Crossplane, Anthos and ArgoCD

Crossplane: Implementing a Gitops approach for your infrastructure

Crossplane is an open source tool for managing cloud resources with Kubernetes and its logic. Crossplane allows you to centralize and simplify the deployment of your cloud resources, to apply the same security logic as in your Kubernetes workloads, and to implement a full Gitops approach even for your cloud services

Crossplane - The cloud-native control plane framework

Argocd : Automate your deployments with confidence with Argo CD

ArgoCD is an open source tool that allows you to implement the continuous deployment part of your Kubernetes workloads with a declarative approach.

ArgoCD provides a multitudes of functionalities in the management of environments, in the various strategies of deployment (canary release, A/B Testing, rolling update),

ArgoCD. also provides a web interface to manage and view the status of your deployments.

Argo CD - Declarative GitOps CD for Kubernetes

Anthos : Simplify the management of your Kubernetes workloads with Anthos

Anthos is a hybrid, multi-cloud platform that allows you to deploy and manage a large fleet of Kubernetes clusters, ensuring consistency, compliance, observability and management at scale.

Anthos provides features such as:

1. Anthos clusters ( GKE — Anthos on AWS — Anthos on Azure — Anthos on vmware — Anthos edge — anthos bare metal )
2. Anthos config management , it is a Gitops oriented tool allowing to manage from one or more git repositories the configuration and security ( OPA gatekeeper ) of all your clusters wherever they are deployed
3. Anthos service mesh, as its name indicates it is a managed service mesh based on istio, deployable anywhere
4. Anthos connect gateway offers you the possibility to simplify the authentication and authorization to your clusters or to deploy them via Google cloud authentication.

Focus on Crossplane: operation and specificities

Crossplane is a Kubernetes-based infrastructure management tool that enables declarative and automated deployment and management of cloud resources. Unlike Terraform, Crossplane uses Kubernetes APIs and principles to define and manage infrastructure resources, providing a unified interface for developers and operators.

Key terms used in Crossplane include:

1. Provider: Providers are extensions to Crossplane that allow you to deploy and manage infrastructure resources for different cloud service providers, such as AWS, GCP, Azure, etc. Documentation
2. Custom Resource Definition (CRD): CRDs are extensions to the Kubernetes APIs to define new types of infrastructure resources. Documentation
3. Managed Resources: These are instances of CRDs that represent provider-specific infrastructure resources. Documentation
4. Composition: A Crossplane composition is a user-defined template that describes how to combine and configure managed resources to create a custom, reusable infrastructure solution. Documentation
5. Claim: Compositions represent requests by applications to use infrastructure resources. Documentation
6. CompositeResourceDefinition (XRD): XRD is an extension to the Kubernetes APIs in Crossplane that defines a new type of composite resource, allowing custom abstractions for infrastructure resources to be created.

Why Crossplane ?

We want to provide more flexibility and simplicity to developers while having control over security and architecture.

Offering developers the ability to consume cloud resources like any other Kubernetes resource but this time for their infrastructure needs, Crossplane is exactly what we need.

Let’s imagine that a product team needs to deploy an application on GKE with the following requirements for this application :

1. A Cloud storage bucket
2. Google service account
3. Policy binding of this service account with the role roles/storage.objectViewver
4. The Workload identity setup for the GSA + the KSA provided by the product team.

As a platform admin I want to provide a simple interface to my developers to access these cloud resources on demand.

To do this, we will use Crossplane features to define a new Kubernetes object called GkeWiBucket.

First, we will create a Crossplane composition that defines a logical sequence of steps like you could do with terraform.

Crossplane Docs · v1.10 · Composition

This is only an example of code, which is not the most optimal :

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: gke-wi-bucket
spec:
  compositeTypeRef:
    apiVersion: dev.onepoint.org/v1alpha1
    kind: XGkeWiBucket
  resources:
    - name: crossplane-bucket-plateform
      base:
        apiVersion: storage.gcp.crossplane.io/v1alpha3
        kind: Bucket
        metadata: 
          name: test
        spec:
          forProvider:
            location: US
          providerConfigRef:
            name: default
      patches: 
      - fromFieldPath: spec.location
        toFieldPath: spec.forProvider.location
      - fromFieldPath: spec.bucketname
        toFieldPath: metadata.name
      
    - name: crossplane-bucket-sa
      base:
        apiVersion: iam.gcp.crossplane.io/v1alpha1
        kind: ServiceAccount
        metadata:
          name: "{{ .inputs.googlesa }}"
        spec:
          forProvider:
            displayName: test
          providerConfigRef:
            name: default
      patches:
      - fromFieldPath: spec.googlesa
        toFieldPath: metadata.name
      - fromFieldPath: spec.googlesa
        toFieldPath: spec.forProvider.displayName

####################@
    - name: crossplane-bucket-sa-role-binding
      base:
        apiVersion: iam.gcp.crossplane.io/v1alpha1
        kind: ServiceAccountPolicy
        metadata:
          name: crossplane-bucket-sa-role-binding
        spec:
          forProvider:
            serviceAccountRef:
              name: my-top-gsa
            policy:
                bindings: 
                - members:
                    - serviceAccount:vincent-ledan-378413.svc.id.goog[test/vincent-test]
                  #  - serviceAccount:my-top-gsa@vincent-ledan-378413.iam.gserviceaccount.com
                  role: roles/iam.workloadIdentityUser

          providerConfigRef:
            name: default

      patches:
        - fromFieldPath: spec.googlesaid
          toFieldPath: spec.forProvider.policy.bindings[0].members[0]
        - fromFieldPath: spec.roleserviceaccount
          toFieldPath: spec.forProvider.policy.bindings[0].role    

    - name: crossplane-bucket-sa-role-binding-role
          base:
            apiVersion: iam.gcp.crossplane.io/v1alpha1
            kind: ServiceAccountPolicy
            metadata:
              name: crossplane-bucket-sa-role-binding-role
            spec:
              forProvider:
                policy:
                    bindings: 
                    - members:
                        - serviceAccount:vincent-ledan-11111.svc.id.goog[test/vincent-test]
                      #  - serviceAccount:my-top-gsa@vincent-ledan-378413.iam.gserviceaccount.com
                      role: roles/storage.objectViewver
    
              providerConfigRef:
                name: default
    
          patches:
            - fromFieldPath: spec.googlesaid
              toFieldPath: spec.forProvider.policy.bindings[0].members[0]
            - fromFieldPath: spec.roleserviceaccount
              toFieldPath: spec.forProvider.policy.bindings[0].role

Then create a CompositeResourceDefinition which will define the schema of the object to present to the developers, here is an example:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xgkewibuckets.dev.onepoint.org
spec:
  group: dev.onepoint.org
  names:
    kind: XGkeWiBucket
    plural: xgkewibuckets
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              location:
                type: string
              bucketname:
                type: string
              googlesa:
                type: string
              saname:
                type: string
              sanamespace:
                type: string
              roleserviceaccount:
                type: string
              projectid:
                type: string
              googlesaid:
                type: string
            required:
              - location
              - bucketname
              - saname
              - sanamespace
              - googlesa
              - roleserviceaccount
              - projectid
              - googlesaid
  claimNames:
    kind: GkeWiBucket
    plural: gkewibuckets

Then you can present the following Kind to your developers:

apiVersion: dev.onepoint.org/v1alpha1
kind: GkeWiBucket
metadata:
  name: claimed-gke-bucket
  namespace: test
spec:
  location: "US" ### region bucket 
  bucketname: "vincentledan-test-my-app" ###bucket name
  googlesa: "my-top-gsa" ### service account google IAM 
  roleserviceaccount: "roles/storage.objectViewer" ### role du service account 
  saname: vincent-kube-sa ### nom du kubernetes account à lié 
  sanamespace: default ### namespace du Kubernetes service account 
  projectid: vincent-ledan-1111 ### google project id 
  googlesaid: my-top-gsa@vincent-ledan-378413.iam.gserviceaccount.com ### nom complet du googlesa 

We have just seen an example concerning the needs of a team of developers and how to provide as a service cloud services from Kubernetes.

The logic above considers that we were in a Multi-tenant cluster, the Composition and CompositeResourceDefinition objects are scoped cluster and the GkeWiBucket claim is scoped namespace which allows to provide rights to a team in this namespace and that they are autonomous to deploy their applications and their cloud infrastructure in the same way.

Practical guide: Assembling Crossplane, Anthos and ArgoCD for an optimized platform

Today, the goal is to extend this approach to the entire organization.

The proposed architecture aims to simplify application deployment, primarily on Kubernetes, strengthen security with Anthos Config Manager and Policy Controller, and take a GitOps approach to cloud resource management and deployment using Crossplane. We will review the key steps in building this platform.

Step 1: Control plane initialization

As a platform team we create a GKE autopilot cluster with Anthos config management activated. ACM will take care of configuring Crossplane and ArgoCD via a repository that you will have made available to it.

Then we will for example create a Crossplane composition allowing to create and configure a GKE/Anthos cluster and to use a claim as previously used in a namespace dedicated to infrastructure teams.

Concerning the registration of GKE clusters with ArgoCD, it is possible to centralize and not to expose secrets thanks to the implementation of Connect Gateway Anthos.

Connect Gateway with ArgoCD | Google Cloud Blog

At this stage your control plane is configured as follows:

1. authentication between GKE and ArgoCD is done through the Anthos connect gateway
2. Argocd is installed and ready to use
3. 1 Argocd project is configured and contains a crossplane composition to deploy the GKE infrastructure.

Step 2: Configure Developer Access

We will assume that we have 2 teams deploying applications in 2 regional clusters.

The platform team will configure RBACs to allow team1 to deploy in the namespace: product1-team and team2 in product2-team.

The platform team will also configure two specific projects in ArgoCD and configure the OIDC groups to allow the right rights for the right teams in the right projects.

Each Argocd project will have clusters assigned to it:

1. Project team1 will only see its clusters APP1 and APP2
2. Project team2 will only see its clusters APP3 and APP4

Since version ArgoCD 2.5, it is now possible to deploy Kinds Application directly in certain namespaces which gives more autonomy and does not oblige the users to deploy in the namespace system of ArgoCD.

Step 3: Strategy and Configuration of the different Gitlab repositories

Let’s take the example of the configuration for the APP1 application that must be deployed in the APP1 cluster.

The team in charge of deployment will create a repository containing a CI/CD for managing the helm chart but also for deploying via a script or other part /Argocdconfigs which contains 2 Kinds Application, one to deploy the infrastructure via crossplane and one to deploy the application in the target cluster.

When these two applications have been applied on the cluster control plane of config, ArgoCD will sync and apply the helms chart, the infra part will be deployed in the team1 namespace locally and Crossplane will take over to create and manage the infrastructure, and the application will be deployed directly in the remote cluster.

At this point the following architecture will be in place.

Step 4: Secure all your deployments with Crossplane and Policy controller

Finally we can focus on the Anthos config management part and more particularly on the Policy controller feature which is the implementation of OPA Gatekeeper in google cloud.

Since all resources, both infrastructure and application, are now managed by Kubernetes, we can easily implement compliance rules as code for Kubernetes workloads as well as for our custom resources.

For example if we want to control the name of the GCP account services created by our new GkeWiBucket resource, typically the value of googlesa

apiVersion: dev.onepoint.org/v1alpha1
kind: GkeWiBucket
metadata:
  name: claimed-gke-bucket
  namespace: test
spec:
.............................
  googlesa: "my-top-gsa" ### service account google IAM 
.............................

We can easily create the following OPA rule :

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: deny-admin-sa
spec:
  crd:
    spec:
      names:
        kind: DenyAdminSA
      validation:
        openAPIV3Schema:
          properties:
            spec:
              properties:
                googlesa:
                  type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package deny_admin_sa

        deny[msg] {
            input.kind == "GkeWiBucket"
            input.spec.googlesa.startswith("admin.mycompany")

            msg := sprintf("Deployment of %v denied because spec.googlesa starts with 'admin.mycompany'", [input.metadata.name])
        }

---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: DenyAdminSA
metadata:
  name: deny-admin-sa
spec:
  match:
    kinds:
      - apiGroups: ["dev.onepoint.org"]
        kinds: ["GkeWiBucket"]
  parameters:
    googlesa: "admin.mycompany"

Conclusion

In conclusion, adopting a GitOps approach and using tools such as Crossplane, Anthos and ArgoCD can simplify cloud infrastructure management and accelerate the implementation of modern platforms. With the rise of platform engineering practices, we can expect continued growth of these solutions and an evolution of infrastructure management methods in the coming years. Organizations that adopt these innovative approaches will be better positioned to meet the challenges of the future and take full advantage of the benefits modern technologies offer.

I would like to emphasize that this architecture is widely debated, this article is intended to show an alternative approach to terraform and more traditional deployments.