Use Advanced Cluster Management(ACM) templates to manage Users,Groups and Namespaces.

On-boarding users /teams onto Kubernetes can sometimes be difficult to achieve. There are multiple tools that can be managed for such use cases:

• A static configuration tool like Kustomize can be used, but it does not provide templates to allow dynamic configuration/processing.
• A tool like Helm can also be used as it does provide excellent template functionalities, but it can be very complex to use for such a purpose.
• Other tools like HNC,NameSpace Configuration Operator are also useful based on your use case.

In my opinion Advanced Cluster Management(upstream: Open Cluster Management) provides a Governance and Policy Framework that can provide significant benefits for on-boarding users and generating resources. Some of the reasons I think so include:

• ACM provides advanced template functionality based on Go Templates. Here’s a great guide on using ACM templates.
• A Gitops approach with ArgoCD can be adopted with ACM as the templating engine.
• ACM can query the status of objects in a kubernetes cluster and use that status as a dependency for the creation of other objects.

Let’s look at some practical use cases:

• Secret Distribution: A very common problem is secret distribution in Kubernetes applications and infrastructure. ACM templates can be used to lookup the value of an existing secret across namespaces or even clusters. Allowing an admin to create a secret that can be shared and managed globally.

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-global-secret
  namespace: global-policies
  annotations:
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/standards: NIST SP 800-53
spec:
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-global-secret
        spec:  
          object-templates-raw: |
            {{- range (lookup "v1" "Namespace" "" "" "deployer=kustomize-global").items }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: Secret
                data:
                  username: '{{hub fromSecret "global-policies" "global-secret-for-every-namespace" "username" hub}}'
                  password: '{{hub fromSecret "global-policies" "global-secret-for-every-namespace" "password" hub}}'
                metadata:
                  name: global-secret-for-every-namespace
                  namespace: '{{ .metadata.name }}'
                  labels:
                    app.kubernetes.io/name: "base_config"
                type: Opaque
            {{- end }}
          pruneObjectBehavior: DeleteIfCreated
          remediationAction: enforce
          severity: medium
  remediationAction: enforce

Looking at the example above the key parts are:

• ‘object-templates-raw:’ means we will be creating an advanced template that can support range and if else functions.
• ‘{{- range (lookup “v1” “Namespace” “” “” “deployer=kustomize-global”).items }}’ means we are going to loop through all objects of type namespace that are labeled with deployer=kustomize-global and apply the template(secret yaml) below that line for every object(namespace) in that loop.
• ‘username: {{hub fromSecret “global-policies” “global-secret-for-every-namespace” “username” hub}}’ means we are going to lookup the value of key username from an existing secret called ‘global-secret-for-every-namespace’ in the ‘global-policies’ namespace. We also do the same for value secret.
• ‘namespace: ‘{{ .metadata.name }}’ means we are going to take the name of the object we looped through, and since we are looping through namespaces that would be the namespace name.
• The final effect of this policy is to create a secret in every namespace labeled with deployer=kustomize-global. And in this example this policy would be applied on every cluster in ACM, as we set it for our global clusterset.

User Object Creation and Onboarding: A probably more important use case is how to create Kubernetes resource’s for our users based on group memberships or other requirements. In essence what the policy below will do is look up which users are in team-a, and create a Namespace,RoledBinding and ResourceQuota for every user in team-a. This provides us the power to manage our Users through our IDP without having to statically define them, yet dynamically use ACM to look up group memberships and other traits and then use those traits to create resources for users or groups.

# Since this policy uses a user name and searches if a string contains a substring versus an exact match having short usernames will create a problem.
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-team-a-devspaces
  namespace: global-policies
  annotations:
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/standards: NIST SP 800-53
spec:
  dependencies:
     <... Our main policy depends on policies defined here to be succesful before it can be applied ..>
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-team-a-devspaces-ns #Create a developer namespace for all users in team-a
      spec:  
        object-templates-raw: | 
          {{- range (lookup "user.openshift.io/v1" "User" "" "" "").items }}          
          {{- if contains .metadata.name ((lookup "user.openshift.io/v1" "Group" "" "team-a" "").users | join "_") }}
          - complianceType: musthave
            objectDefinition:
              kind: Namespace
              apiVersion: v1
              metadata:
                name: {{ .metadata.name }}-devspaces
                labels:
                  app.kubernetes.io/part-of: che.eclipse.org
                  app.kubernetes.io/component: workspaces-namespace
                  deployer: "kustomize-global"
                  team-a-devspace: "true"
                  devspace-owner: {{ .metadata.name }}
                annotations:
                  che.eclipse.org/username: {{ .metadata.name }}
                  project-owner: team-a
          - complianceType: musthave
            objectDefinition:
              apiVersion: rbac.authorization.k8s.io/v1
              kind: RoleBinding
              metadata:
                name: {{ .metadata.name }}-admin-devspaces
                namespace: {{ .metadata.name }}-devspaces
              roleRef:
                apiGroup: rbac.authorization.k8s.io
                kind: ClusterRole
                name: admin
              subjects:
              - kind: User
                apiGroup: rbac.authorization.k8s.io
                name: {{ .metadata.name }}
          - complianceType: musthave
            objectDefinition:
              apiVersion: v1
              kind: ResourceQuota
              metadata:
                name: {{ .metadata.name }}-devspaces-quota
                namespace: {{ .metadata.name }}-devspaces
              spec:
                <...ResourceQuota Object ...>
          {{- end }}
          {{- end }}
        pruneObjectBehavior: DeleteIfCreated
        remediationAction: enforce
        severity: medium
  - extraDependencies:
    - apiVersion: policy.open-cluster-management.io/v1
      compliance: Compliant
      kind: ConfigurationPolicy
      name: policy-team-a-devspaces-ns
    objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-team-a-devworkspace #Create a devworkspace for each user in team-a
      spec:  
        object-templates-raw: |
          {{- range (lookup "v1" "Namespace" "" "" "team-a-devspace=true").items }}
          - complianceType: musthave
            objectDefinition:
              apiVersion: workspace.devfile.io/v1alpha2
              kind: DevWorkspace
              metadata:
                name: nodejs-mongodb
                namespace: {{ .metadata.name }}
              spec:
              <...devworkspace object...>
          {{- end }}
        pruneObjectBehavior: DeleteIfCreated
        remediationAction: enforce
        severity: medium

Looking at the example above the key parts are:

• ‘ dependencies:’ Our policy depends on other policies that make sure other resources are created and exist before this policy will be applied.This helps us provide ordering guarantees.
• ‘object-templates-raw:’ means we will be creating an advanced template that can support range and if else functions.
• ‘{{- range (lookup “user.openshift.io/v1” “User” “” “” “”).items }}: This means we are going to start our first loop that loops over every user object defined in the cluster.
• ‘{{- if contains .metadata.name ((lookup “user.openshift.io/v1” “Group” “” “team-a” “”).users | join “_”) }}’: We are going to create a nested loop that loops over the user’s that are members of team-a. This creates the effect of checking through every user in the cluster if they are members of team-a. If they are members the rest of the policy is now applied.
• ‘complianceType: musthave’: means we expect what ever object yaml defined in the ‘objectDefinition’ to exist in the cluster.
• Looping through the objects we will create a namespace named ‘{{ .metadata.name }}-devspaces’ which would resolve to the username-devspaces.
• We will create a RoleBinding called {{ .metadata.name }}-admin-devspaces. This would create be username-admin-devspaces in our previously created namespace.
• And then we create a ResourceQuota called ‘{{ .metadata.name }}-devspaces-quota’ in the same created namespace.
• A second policy in the example will lookup the namespace we created for every user based on a label and create an openshift devspaces instance for them.(‘{{- range (lookup “v1” “Namespace” “” “” “team-a-devspace=true”).items’)

One more advantage of using ACM for this use case would be ACM’s UI. We can use it’s UI to track ourpolicies and view the actions they applied and the status of resources that were created. As an example here’s a snapshot of the above policy, which created a Namespace, RoleBinding and ResourceQuota for every user in team-a.

ACM View for Policy Created Resources

The code repo for these examples can be found here:

GitHub - MoOyeg/acm-onboarding-examples: Use Red Hat Advanced Cluster Management(RHACM) to onboard…

Hopefully this was useful!