Walkthrough: Crossplane for your IDP’s landing zone

You may have heard about Crossplane as an IaC tool allowing you to spin up all your infrastructure from the ground up. In this post I will not use Crossplane to build everything, but I will focus on things that are created to allow your customers to use your Platform.

crossplane.io

When you build an Internal Developer Platform (IDP) you quickly start realising that there are several things that need to be done for every new project (coming from your customers) to onboard on your Platform.

The list of things to do will vary depending on the number of services that you provide and the stack you use.

In this blog post I will show you how Crossplane can be a very appropriate solution to hide all the complexity of this onboarding phase while offering a very simple interface for end users.

The environment I’ll use in this walkthrough will be based on the following AWS services:

• AWS EKS
• AWS ECR for the container registry
• AWS Secrets Manager and External Secret Operator (ESO) to use native Kubernetes Secrets with secrets stored in a vault with IAM.

I choose these services because they represent the most common components that one can find in an Internal Developer Platform; a Kubernetes engine, a container registry and a secret vault.

There is no doubt a Platform is much more than that, but these components are enough to showcase what Crossplane can help you achieve.

I will not describe how I span up the EKS cluster on AWS as there are a lot of available resources on the web already. It can be with any tool, it won’t change anything for our Crossplane implementation for the onboarding phase.

Note: This post will walk you through the implementation and the resulting code can be found in this GitHub repository.

The problem to solve

A new project is born in the company, a team is built around it and it will be deployed on my Platform.

Based on the list of services that I use, I can list mandatory things that I’ll have to do to allow the new team to use my Platform:

• Create a Kubernetes namespace to let them deploy their app
• Deploy a SecretStore in the namespace to allow the External Secret Operator to read secrets in Secrets Manager and create Secrets in the namespace. I probably want to restrict what secrets can be retrieved by a SecretStore so that a project cannot fetch the secret from another.
• Create permissions for the team to push and pull images in AWS ECR in repositories that must contain the project name as a prefix.

What is important now is to ask ourselves how all these onboarding tasks can be done without being a struggle for anyone.

First, you don’t want your Platform team to be triggered every time a project needs to be onboarded. It would create noise for this team and create interruptions that could easily be avoided. Moreover, this is typically the toil you want to eliminate.

While we obviously need to automate all these tasks somehow, it is important that the interface for your customers remains as simple as possible.

Let’s keep in mind that the primary role of a Platform team is to reduce cognitive load for developers, so we should avoid asking them to learn the basis of Terraform, Ansible or another tool and come update our code somewhere.

So we must find a way to allow the project team to trigger the onboarding process in a fully autonomous way and keep it dead simple.

Crossplane to the rescue

Crossplane is a framework that allows you to create your own Custom Resource Definitions (CRD) without actually coding an operator. It allows creating Compositions that are declarative sets of resources created by providers.

For our use case, using Crossplane could help us hide all the complexity of the onboarding behind a Custom Resource that would trigger the creation of all the required resources in AWS and Kubernetes when the Custom Resource is claimed.

I will walk you through an implementation of Crossplane (v1.13.2) that will allow this resource to hide the complexity of the whole onboarding phase tasks.

Installing Crossplane

Installing Crossplane is simply about installing a Helm chart in your cluster. You can follow the quickstart available on the Crossplane website. The chart will deploy the Crossplane controller(s) and a bunch of Custom Resources that allows configuring Crossplane.

The next phase of the installation consists in installing the required providers and to configure them. For my use case, I will need to create resources in AWS and in Kubernetes so I need to install and configure the AWS Provider and the Kubernetes Provider.

Installing the providers is as simple as deploying the following manifest:

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-iam
spec:
  package: xpkg.upbound.io/upbound/provider-aws-iam:v0.37.0
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.9.0

For the configuration phase, I used the quickstart methods as well (AWS and Kubernetes) which are not production ready but are enough for me here.

Create the custom resource

With Crossplane and the required providers installed, I can start working on defining my custom resource.

As I stated at the beginning of the article, I want to hide the complexity of creating several resources in AWS and in Kubernetes behind a new Kubernetes resource (CRD) that I want to call a LandingZone.

This resource would be very simple and would require only one parameter for my example:

---
apiVersion: platform.suited.sh/v1alpha1
kind: LandingZone
spec:
  region: eu-west-2

Note: The only required information will be the region where the AWS resources shall be created.

Crossplane allows you to create a custom resource by creating a CompositeResourceDefinition, a CRD installed in your cluster when you install Crossplane.

A CompositeResourceDefinition (or “XRD” in the Crossplane world) is the object that will allow you to declare the OpenAPIv3 schema of your new resource.

The XRD for my resource looks like this:

# landing-zone-xrd.yaml 
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: landingzones.platform.suited.sh
spec:
  group: platform.suited.sh
  names:
    kind: LandingZone
    plural: landingzones
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              region:
                type: string
    served: true
    referenceable: true

Note: When writing this XRD I chose to create an API Group named platform.suited.sh. API Groups define a collection of related API endpoints. “suited.sh” is my company and you can replace this with yours.

For more information about the XRD available fields you can refer to Crossplane documentation.

At this stage I can already apply this manifest and verify that I can query my cluster to list all available XRDs that should return my newly created XRD:

$ kubectl apply -f landing-zone-xrd.yaml 
compositeresourcedefinition.apiextensions.crossplane.io/landingzones.platform.suited.sh created
$ kubectl get xrd
NAME                              ESTABLISHED   OFFERED   AGE
landingzones.platform.suited.sh   True          True      3s

At this stage I have created the custom resource I want to use to trigger the creation of all my resources but I have nothing telling Crossplane what to do when someone claims a LandingZone.

To tell Crossplane what to do, I have to provide a template describing what infrastructure to deploy. Crossplane calls this template a Composition.

Create the Composition

Before implementing the whole resources in the Composition, I will start with a very simple implementation that only creates the namespace.

# landing-zone-composition.yaml
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: suited-landing-zone
spec:
  resources:
    - name: namespace
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          forProvider:
            manifest:
              apiVersion: v1
              kind: Namespace
          providerConfigRef:
            name: kubernetes-default
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.name
          toFieldPath: spec.forProvider.manifest.metadata.name
  compositeTypeRef:
    apiVersion: platform.suited.sh/v1alpha1
    kind: LandingZone

Notes:

- The “region” parameter will be unused here as I am not configuring any AWS resource yet.

- I want the namespace to be named after the project name that I expect people to provide as the name of the LandingZone.
By default crossplane would use this value but add a hash which would make it non deterministic for my customers.
I can override the resource name through patches section and have a deterministic name.

- We provide a provider configuration through the providerConfigRef that represents the name of a ProviderConfig object that we created during the provider installation phase. This means that a Composition can create resources using different provider configurations, so potentially different clusters, AWS accounts and so on.

- What makes the glue between the XRD and the Composition is what is defined in the compositeTypeRef at the end of the Composition where information about my XRD is provided.

For more information about the Composition fields you can refer to Crossplane documentation.

At this stage I can deploy this Composition in my cluster:

$ kubectl apply -f landing-zone-composition.yaml 
composition.apiextensions.crossplane.io/suited-landing-zone created

And verify we see the Composition in the cluster:

$ kubectl get composition
NAME                  XR-KIND       XR-APIVERSION                 AGE
suited-landing-zone   LandingZone   platform.suited.sh/v1alpha1   30m

All good !

Testing for the first time with the simplified composition

Now that I have a custom resource (the XRD) and its implementation (the Composition in its dummy form) created and deployed, I can test that everything is working as expected.

Let’s create a claim for a Landing Zone:

# landing-zone-claim.yaml
---
apiVersion: platform.suited.sh/v1alpha1
kind: LandingZone
metadata:
  name: awesome-project
spec:
  # At this stage, this information is not used
  region: eu-west-2

And let’s deploy it in our cluster. If things are working properly, after the creation of my LandingZone resource I should see an awesome-project namespace created shortly after:

$ kubectl apply -f landing-zone-claim.yaml
$ kubectl get landingzones
NAME                     SYNCED   READY   COMPOSITION           AGE
my-sample-landing-zone   True     True    suited-landing-zone   55s

If the LandingZone is marked “synced” and “ready”, it means I should find my namespace as well:

$ kubectl get ns
NAME                STATUS   AGE
awesome-project     Active   2m18s         # yeah !
crossplane-system   Active   46h
default             Active   46h
kube-node-lease     Active   46h
kube-public         Active   46h
kube-system         Active   46h

It’s a win !

If you are curious, you can check the events for the LandingZone object and see that we have some details about what happened:

$ k describe landingzones my-sample-landing-zone
[...]
Events:
  Type    Reason                   Age                    From                                                             Message
  ----    ------                   ----                   ----                                                             -------
  Normal  CompositionUpdatePolicy  6m57s                  defined/compositeresourcedefinition.apiextensions.crossplane.io  Default composition update policy has been selected
  Normal  ComposeResources         6m57s (x3 over 6m57s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  Composed resource "namespace" is not yet ready
  Normal  SelectComposition        56s (x11 over 6m57s)   defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully selected composition
  Normal  ComposeResources         56s (x11 over 6m57s)   defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully composed resources

Based on this output and particularly this part:

Composed resource “namespace” is not yet ready” you can guess that the LandingZone will only be reported “ready” if all the resources from the composition are successfully created.

You can guess that the LandingZone will only be reported “ready” if all the resources from the Composition are successfully created.

Now that we know that the Crossplane mechanism is working with this very simple Composition, we can write the complete one and start introducing the AWS resources.

Implement the full Composition

We can start by adding the resources required for the External Secrets Operator.
This is an interesting part because it mixes resources in AWS and Kubernetes:

• The SecretStore custom resource (Added in the cluster when installing externa-secrets through Helm)
• A ServiceAccount with the IRSA annotations allowing to associate it with an IAM Role
• The IAM Role and associated permissions required for the SecretStore to work

It is a good example because usually deploying resources in Kubernetes and creating resources in AWS are not using the same tooling (eg: ArgoCD/FluxCD vs Terraform).

After adding the SecretStore related resources, we can add the registry related resources.

As the Platform builder, I’d like my customers to be able to:

• Push/Pull images to the registry from a CI/CD pipeline
• Allow them to create any number of images for their project from the CI/CD Pipeline. With AWS ECR it means they need to be able to create repositories. I will restrict permissions based on the repository name that will contain the project name as a prefix.
• Pull images from their local machine.

I will represent the CI/CD permissions by a dedicated role and the access from the team members by an IAM Group. Of course I want everything to be created automatically by my LandingZone.

I will not post all the YAML we need to add in the Composition here because it would be too much information (~150 lines just for the SecretStore part), but you can find the full implementation of the Composition in this GitHub repository.

Offering self-service to the Platform users

We now have a complex set of resources created in different places (AWS and Kubernetes) managed by Crossplane which in return offers a very simple interface: a yaml file.

What we did not talk about yet is how our customers could use this interface in an autonomous way.

Let me list a small list of options:

- The GitOps option

If your Platform includes a GitOps solution like ArgoCD or FluxCD, you can use these tools to offer self service for the onboarding phase.

• You can dedicate a repository for the custom resource you’ll create and your customer would simply have to create a merge/pull request that adds the yaml in a directory.
• What is important here is to make the experience simple:
• Properly document everything so people don’t have to search for hours how to use the repository
• Have a CI workflow that makes all the required checks so that typos and invalid schemas are detected early

- The CLI option

Having an internal CLI become more and more common these days. Having a sub-command allowing you to interact with your platform could be a very interesting approach since your customer will already be familiar with the CLI usage.
The experience could be as simple as :

# opinionated cli example
$ your-cli platform landing-zone create --name awesome-project

Having an API endpoint actually writing the resource YAML in a repository could allow you to keep the GitOps deployment behind the scenes. But developing the API is an additional work to do.

- The Web UI option

Whether it is custom made or based on an existing solution like Backstage, Port or Qovery. It should be possible to allow these solutions to offer a web UI and a form that ultimately allows the creation of the XRD in the Cluster hosting Crossplane.

If the GitOps option could be pretty straight forward if you already have it in your stack, the CLI and Web UI options would offer a better UX in my opinion. They would also require dedicated blog posts to enter the implementation details !

Adding some guardrails

This new resource creation process currently has no guardrails around what the end user can input in resource definition. With the implementation as is, the name provided by the end user is directly re-used to create the namespace for example and one could enter “kube-system” or another name that you probably would like to protect.

There are solutions like Gatekeeper, Kyverno and Datree (eg) that would allow making sure that what will be sent to Crossplane is safe.

Be sure to have a system acting as guardrails in your workflow to make sure that self-service cannot trigger issues for you afterward.

Note: Gatekeeper and Kyverno will block the “apply” operation in case of policy violation, which is already late in the issue detection if you are offering self-service to your customers.
With self-service you want people to get the feedback around an issue as soon as possible, like in the Pull Request with a GitOps workflow, or from the CLI/Web UI for these respective options.

This is important to let the end user be able to fix the issue immediately instead of staying blind and waiting for something that will never happen.

End note

Crossplane is a very exciting product with an ecosystem of providers available through Upbound Marketplace.

The major Cloud Providers are of course supported but you will also find providers for tools and use cases that you may have in your company:

• If you use HashiCorp Vault instead of AWS SecretsManager, there is a provider for you
• If you offer database as a service, there is a provider to automate the users creation
• If you use Grafana, there is a provider that will help you init a folder and dashboard for the team.
• …

The library will grow along with the community which promises a lot of interesting integrations and possibilities.

The Crossplane providers are now generated from Terraform providers through Upjet (a code generator) and the 3 majors cloud providers Crossplane providers are built with Upjet.

It means that if you don’t find a Crossplane provider and you know a Terraform provider exists, you can potentially generate the Crossplane provider yourself and use Crossplane !

Note: As of today, the HashiCorp announcement around the change of licensing to the BSL license is not impacting the providers.

If you are struggling with the way your customers interact with your Platform, I would definitely recommend having a look at Crossplane as a means to make the experience very simple for your end users !