Writing a Crossplane Composition Function

8 min read6 days ago

Hello there 👋🏻 I’m grateful for your time and hope this article will deliver valuable information!

I want to approach today's topic by discussing how the Crossplane Composition Functions helped me gain more flexibility in creating and managing compositions.

The challenge

I’ve been using Crossplane for quite some time now, mainly to empower engineers to create the required dependencies for the services they operate using already-known technologies like YAML, Kubernetes, and Helm.

The challenge consists of updating the existing flow to create GCS instances, which had a critical flaw. It used a single Service Account to impersonate and authorise all services, posing a significant security threat that demanded immediate attention.

Solution

The solution was straightforward: adjust the existing Crossplane composition to create an SA per GCS instance with restricted access. This means that the following resources had to be included in the already existing composition:

ServiceAccount — Used for Authorization;
ServiceAccountKey — Required for the SDK used by the service to authenticate;
BucketIamMember — Assigning permission to the Service Account;

The first issue appeared regarding the name of the service account. Since the composition will be reused in the cluster, the names must be unique. Some logic to generate a unique name for each instance was required.

The default method of defining compositions was not helpful because it limits you to the go templating that can be used before the installation time to implement any logic. Still, this requirement involves analysing names and creating unique IDs.

With this first blocker, I tried to find solutions, and that’s when I learned about Composition Functions.

A summary of composition functions: Instead of creating dependencies for the Crossplane team to implement a custom feature you need for the composition, functions enable us to write the logic we need, taking advantage of High-Level programming languages like Go or Python. The functions run in a pipeline-like environment where the output of a step serves as the input for the new one.

Implementation

I’ve started the function creation process following the documentation. There are two ways to init a project:

cloning the desired function template repo from GitHub;
using the CLI;

crossplane beta xpkg init function-bucket-permission function-template-go -d function-bucket-permission

The template contains some boilerplate code used for internal communication in the Crossplane ecosystem. Still, the main things that you need to take care of initially are covered in the documentation:

Update the input/v1beta1 directory to reflect the desired input. The project name acted as an input for the function to compose the service account emails;

type Input struct {
 metav1.TypeMeta `json:",inline"`
 metav1.ObjectMeta `json:"metadata,omitempty"`
 
 // Example is an example field. Replace it with whatever input you need. :)
 ProjectName string `json:"projectName"`
}

Add the required go modules. I am working with GCP Managed Resource, so I had to include the GCP Provider using the following commands:

go get github.com/upbound/provider-gcp/apis/cloudplatform/v1beta1@v1.0.0
go mod tidy

Add code logic to the fn.go file.

The logic was simple; I just wanted to create three resources, where the SA name was the output of an idempotent function generating a unique ID for each combination of bucket name and claim namespace. (data coming from the XR)

Initially, I used a function generating a random ID, but when I got to test the function, I learned a key concept about how functions work: they are called with every reconciliation cycle. So, each time it was called, I would get a new random ID, causing trouble with my setup.

// GenerateID takes two strings and returns a consistent ID of max 10 characters.
func GenerateID(str1, str2 string) string {
 // Create a slice of strings and sort it to ensure consistent order.
 strings := []string{str1, str2}
 sort.Strings(strings)
 // Concatenate the sorted strings.
 concatenated := strings[0] + strings[1]
 // Compute the SHA-256 hash of the concatenated string.
 hash := sha256.Sum256([]byte(concatenated))
 // Convert the hash to a hexadecimal string.
 hexString := hex.EncodeToString(hash[:])
 // Return the first 10 characters of the hex string.
 return hexString[:10]
}

Accessing input data

To access the function input, I used the module defined in the project, called input, which we updated in the beginning:

package main

import (
 "github.com/crossplane/function-bucket-permission/input/v1beta1"
 …
)

func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequest) (*fnv1beta1.RunFunctionResponse, error) {
 rsp := response.To(req, response.DefaultTTL)
 
 in := &v1beta1.Input{}
 if err := request.GetInput(req, in); err != nil {
   response.Fatal(rsp, errors.Wrapf(err, "cannot get Function input from %T", req))
   return rsp, nil
 }
 
 projectName = in.ProjectName
 …
}

Accessing XR data

The function-sdk-go module provides a set of objects and methods, like the request one, to get the composed resource input.

xr, err := request.GetObservedCompositeResource(req)
if err != nil {
 // If the function can't read the XR, the request is malformed. This
 // should never happen. The function returns a fatal result. This tells
 // Crossplane to stop running functions and return an error.
 response.Fatal(rsp, errors.Wrapf(err, "cannot get observed composite resource from %T", req))
 return rsp, nil
}
 
bucketName, err := xr.Resource.GetString("spec.parameters.bucket.name")
namespace, err := xr.Resource.GetString("spec.parameters.namespace")

resourceName := "cp-" + GenerateID(bucketName, namespace)

Creating the managed resources

To create the managed resources, we need to analyse the structure of the API definitions for each resource in the Provider GCP module. The official GitHub repo contains those.

Here's an example of the ServiceAccount API definition.

import (
 ...
 gcpPlatformV1 "github.com/upbound/provider-gcp/apis/cloudplatform/v1beta1"
)

func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequest) (*fnv1beta1.RunFunctionResponse, error) {
  …
  serviceAccount := &gcpPlatformV1.ServiceAccount{
   ObjectMeta: metav1.ObjectMeta{
     Name: resourceName,
   },
   Spec: gcpPlatformV1.ServiceAccountSpec{
     ForProvider: gcpPlatformV1.ServiceAccountParameters{
       Description: ptr.To[string](fmt.Sprintf("Service account to access bucket cp-%s in %s namespace", bucketName, namespace)),
       DisplayName: ptr.To[string](fmt.Sprintf("cp-%s-%s", namespace, bucketName)),
     },
   },
  }
  …
}

Since functions operate in a pipeline-like environment, where the output of one function is the input for the next one, we need to mark that the resource must be created. We need to insert it into the list of desired resources that are part of the req object, but only after converting the object structure in the SDK format can we store the composed resources.

func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequest) (*fnv1beta1.RunFunctionResponse, error) {
…
 desired, err := request.GetDesiredComposedResources(req)
 if err != nil {
   response.Fatal(rsp, errors.Wrapf(err, "cannot get desired resources from %T", req))
   return rsp, nil
 }
 
 // Add the API definitions managed by the Provider GCP module to the known schemes
 gcpPlatformV1.AddToScheme(composed.Scheme)
 
 // Convert the SA to the unstructured resource data format of the SDK
 // uses to store desired composed resources.
 serviceAccountCD, err := composed.From(serviceAccount)
 if err != nil {
   response.Fatal(rsp, errors.Wrapf(err, "cannot convert %T to %T", serviceAccount, &composed.Unstructured{}))
   return rsp, nil
 }

 // Add the new service account to the desired managed resources.
 desired[resource.Name("serviceAccount")] = &resource.DesiredComposed{Resource: serviceAccountCD}
…
}

Setting ProviderConfigs & Writing config to secrets

Crossplane must know which provider should create and manage these resources. Some resources must also write their configurations in Kubernetes secrets so the services running in the cluster can access that data.

These configurations can be achieved using predefined functions, which are defined here.

func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequest) (*fnv1beta1.RunFunctionResponse, error) {
 …
 serviceAccountKey.SetWriteConnectionSecretToReference(&v1.SecretReference{
   Name: fmt.Sprintf("%s-sa-key", bucketName),
   Namespace: namespace,
 })

 // Setting the provider to be used for the managed resource creation
 serviceAccountKey.SetProviderConfigReference(&v1.Reference{
   Name: "provider_name",
 })
 …
}

Returning the response

We must include the desired composed resource list in the res object for the function's response.

// Add the desired managed resources to the response. Crossplane will create
 // these resources after the function returns.
 if err := response.SetDesiredComposedResources(rsp, desired); err != nil {
   response.Fatal(rsp, errors.Wrapf(err, "cannot set desired composed resources in %T", rsp))
 }
 return rsp, nil

Testing

You can add custom tests to check your logic in the fn_test.go file, but since the logic was pretty simple in my use case, I resumed the tests to rendering, where I modified the XR in the example directory to match mine.

Packaging and Publishing

Since the implementation was complete, the next step was to build the function, publish it in a registry, and use it in the cluster.

I’ll leave a template of the code I used inside the pipeline to build and publish the function using a private GitLab registry. The template already includes the CI logic, but it is designed to run in GitHub and publishes the image in the xpkg.upbound.io registry.

There are scenarios where you may not want to do that, and this code acts as an example. (I don’t need to support multiple platforms; that’s why I built only one image version)

image: docker:latest

services:
 - docker:dind

stages:
 - build
 - publish

variables:
 REGISTRY: $CI_REGISTRY/$CI_PROJECT_PATH

before_script:
 - apk add - no-cache curl
 - cd bucket-permission
 # Install Crossplane CLI
 - curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh
 # Login to GitLab Container Registry
 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
 stage: build
 script:
 - echo "Building the runtime container"
 - docker build . - no-cache - platform=linux/amd64 - tag runtime-amd64
 - echo "Building the function"
 - ./crossplane xpkg build - package-file=amd64.xpkg - package-root=package/ - embed-runtime-image=runtime-amd64
 artifacts:
   paths:
   - "./bucket-permission/*.xpkg"
 when: on_success
 expire_in: "30 days"
 only:
 - master
 - tags

publish:
 stage: publish
 dependencies:
 - build
 script:
 - ./crossplane - verbose xpkg push - package-files $(echo *.xpkg|tr ' ' ,) $REGISTRY/bucket-permission:$CI_COMMIT_TAG
 only:
 - tags

NOTE: If you try to build the package locally and use a custom context for the running docker instance, you may confront this error when you use the Crossplane CLI build command:

crossplane: error: failed to get runtime base image options: failed to pull runtime image: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The CLI is looking for a predefined Docker Host path. The solution was to override the path with one of the contexts I was using:

DOCKER_HOST=unix:///Users/<user_name>/.docker/run/docker.sock crossplane - verbose xpkg build \
 - package-root=package \
 - embed-runtime-image=runtime-amd64 \
 - package-file=function-amd64.xpkg

NOTE: Crossplane CLI has the default registry set to xpkg.upbound.io, so in case you want to publish your package to other registries, you need to prefix the repository with the registry type, even if you are logged in locally to that registry.

crossplane xpkg push docker.io/repository:tag

Using the function

To integrate functions in an existing composition, we must first make the function available to the cluster by defining and applying a Function template.

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
 name: bucket-permission
spec:
 package: registry.gitlab.com/org/group/project/package_name:tag
 packagePullSecrets:
 - name: credentials_to_connect_to_the_registry

After that, if we have an existing composition, we need to change it to use the pipeline mode, and the bits that we didn’t integrate into the function need to be managed by a predefined function called patch and transform. This function behaves like the existing way of configuring managed resources in a composition.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
 name: xbucketbackupinstances.clouds.storage.org
 labels:
   crossplane.io/xrd: xbucketbackupinstances.clouds.storage.org
spec:
  compositeTypeRef:
   apiVersion: clouds.storage.org/v1alpha1
   kind: XBucketInstance
   mode: Pipeline
   pipeline:
     # Creating SA, SAKey and BucketIAMMember
     - step: create-authentication-and-authorization-for-buckets
       functionRef:
         name: bucket-permission
       input:
         apiVersion: template.fn.crossplane.io/v1beta1
         kind: Input
         projectName: {{ .Values.providerConfig.projectId }}
     - step: patch-and-transform
       functionRef:
         name: patch-and-transform
       input:
         apiVersion: pt.fn.crossplane.io/v1beta1
         kind: Resources
         resources:
         - name: bucket
           base:
             apiVersion: storage.gcp.upbound.io/v1beta1
             kind: Bucket
             spec:
               providerConfigRef:
                 name: <provider_name>
               deletionPolicy: "Orphan"
               forProvider:
                 uniformBucketLevelAccess: true
           patches:
           - type: FromCompositeFieldPath
             fromFieldPath: spec.parameters.bucket.location
             toFieldPath: spec.forProvider.location

With this, we are almost done!

I am saying almost because if you got here and created an XR using this composition, you will notice that the XR is not ready, even if the resources managed by the function are.

This happens because there is no default mechanism for checking whether the managed resources created by the functions are ready. To fix this, we can either add that logic inside the function or use a predefined function that does that.

- step: check-function-resource-readiness
  functionRef:
    name: auto-ready

You can find more about this here.

Conclusion

I had a blast developing my first Composition Function. This feature gives you endless opportunities when it comes to composition development.

The most exciting finding from this experience was how the maintainer’s team responded to the community's requests: integrating the possibility of using well-known programming languages to build the required flows in the templates is a great way to avoid creating a long waiting time for the adopters for new features to be implemented in the core Crossplane library.