Multi-Cluster Policy Creation, Modification, and Enforcement
The Cloud Engagement Hub has experience working with clients at all stages of their Kubernetes adoption journey, but one thing they all have in common is rapid growth. Whether you’re a small team managing multiple environments (dev/test/prod), a team managing a multi-geographic deployment, or a company managing multiple projects and teams, there are common drivers behind this growth:
As the reasons for deploying multiple clusters in your environment grow, so too can the challenges that arise from doing so without a plan to efficiently manage them. Those challenges likely fit into one of the four categories below:
- Cluster Lifecycle
- Policy & Configuration Lifecycle
- Application Lifecycle
- Observability
This article will focus on the second category, Policy & Configuration Lifecycle, and introduce one tool that could be leveraged to help with this area (and other challenges) of maintaining consistency in your environments. These challenges could include anything from enforcing specific security policies across a cluster fleet, managing configuration drift across clusters, promoting & deploying applications consistently, performing upgrades, to maintaining monitoring and visibility across your environment.
Red Hat Advanced Cluster Management (RHACM) is a tool designed to address challenges such as these in a rapidly growing cluster environment. It can be deployed via an operator from the OperatorHub (link), after which you will then create or import the clusters you want to manage.
Once your cluster fleet is imported, you will be ready to start centrally managing them and applying policies for governance. Let’s start by creating a policy and getting it working.
RHACM Compliancy Policy Creation and Modification
Inside Red Hat Advanced Cluster Management is an area for Governance, Risk, and Compliance. This guide is focused on setting up a basic policy to run against your attached clusters and making simple modifications to it.
Launch Red Hat Advanced Cluster Management for Kubernetes Dashboard
Since I deployed my RHACM instance on an OpenShift cluster in IBM Cloud, I’m going to open the console:
From cloud.ibm.com navigate to your cluster that is hosting your RHACM instance and launch the web console.
From there, you’ll find the RH OpenShift Console page where you will click on the grid icon in the upper right portion of the screen and find the link to launch your RHACM application.
Alternatively, launch from the Networking > Routes multicloud-console Route link:
From here you should see the following landing page:
How Policies Work
Before we create a policy, let’s briefly touch on how they work. Policies are snippets of code (yaml) that can be leveraged to either ‘inform’ or ‘enforce’ a set of conditions on one or more clusters. These policies will apply based on a set of matching labels you configure when creating your policy and apply to your clusters (see note below). When a policy is set to ‘inform’ you will be notified where there is a violation of the policy among the matching labeled clusters for you to investigate and remediate. When a policy is set to ‘enforce’, the policy will attempt to resolve the violation itself automatically if possible.
Note: the labels are defined within RHACM in the Automate infrastructure → Clusters and then 3 dot menu.
Create a Policy
From the Red Hat Advanced Cluster Management for Kubernetes dashboard, select Governance and risk from the home page..
or from the drop down menu:
From there, select the blue ‘Create policy’ button in the upper right:
You will see an empty policy creation screen similar to the one below:
From here, select the following options to follow along:
Name → skip this for now as your selections will populate this field
Namespace → your namespace to deploy the policy to. Here we created an ‘ibm-policies’ namespace
Specifications → These are the list of policies you can select from.
Cluster binding → Here you can specify 1 or more RHACM cluster labels for the policy to apply to.
Standards → Select the name or names of the security policies that your policy should validate. These are user definable via YAML.
Categories → Select 1 or more security categories your policy applies to. These are user definable via YAML.
Controls → Select 1 or more labels for which the policy meets the control security requirements. These are user definable via YAML.
Note: By default, the newly created policy is not enforced and enabled, so there will be informational status messages generated, but no action. You can select to enforce and/or disable the policy via the two checkboxes provided on the Create policy screen.
For demo purposes, I made the following selections if you want to follow along:
Click ‘Create’ in the upper right corner.
You should now see your policy listed in the Policy table, along with the status of your cluster compliance / violation count.
Enforce your Policy
The above policy you deployed looked for any attached clusters labeled with the environment ‘prod’ and looked to ensure they had a ‘prod’ namespace. As you can see, 2 clusters qualified as ‘prod’, but they are not compliant:
Clicking on the policy, under the ‘Placement’ section, you can see the 2 clusters that are marked as ‘Not Compliant’. If you wanted RHACM to automate enforcement of this policy, return to the Governance and Risk page, select the 3 dot menu at the end of the row, and select ‘Enforce’.
After a short wait, you should see your violations resolve as the policy was enforced and those clusters labeled as environment = prod should have a corresponding ‘prod’ namespace created.
Modify / Customize Your Deployed Policy
Let’s say instead of looking for a prod namespace on prod environment labeled clusters, you’d rather ensure a ‘prod’ namespace isn’t created on any RHACM attached clusters tagged as ‘environment=dev’.
Note: I would recommend you change the policy you’re working with back to ‘Inform’ if it’s currently set to ‘Enforce’ just in case you make a mistake.
The above scenario requires us to make a couple changes to our existing policy.
Change the Policy YAML
First, we need to change the condition for the namespace we’re looking for. Select the policy to edit, and select YAML at the top to display the current YAML deployed as the policy. Select the grayed out ‘Edit’ button (yes, this seems to be a bug) to activate it and navigate down to the ‘complianceType’ value, changing it from ‘musthave’ to ‘mustnothave’ and hit Submit.
Change the Placement Rule to Match on ‘dev’ Clusters
Click back to the Details area of the Policy screen where you’ll see you’re again not compliant, as you just changed the policy to not have the ‘prod’ namespace, but you’re still applying your policy to the prod environment clusters you just enforced a prod namespace to.
Click ‘Edit’ on the Placement rule area to bring up the YAML editor:
Here you need to change the value you’re selecting for from ‘prod’ to ‘dev’ and hit submit:
You should now see your Cluster selector match expression change and your list of matching clusters. Give it another few seconds, and their compliance status should update:
This indicates I have one cluster tagged as environment=dev but has a ‘prod’ namespace. I can then go back to the main screen to ‘Enforce’ this policy and have it removed to bring them all into compliance and keep them that way.
Unfortunately, it doesn’t appear you can rename your policies, so if you tweak an original enough, you may want to apply those changes to a new policy with an appropriate name, and remove the modified one. You do that through the 3 dot menu again and select remove:
Note: You can copy and paste an existing policy into the Policy YAML. The values for the parameter fields are automatically entered when you paste your existing policy. You can also search the contents in your policy YAML file with the search feature.
Closing thoughts and resources:
RHACM is one way to get a handle on the types of tasks that either become repetitive or prohibitive when managing multiple clusters.
Here is the location of the Red Hat Advanced Cluster Management policy collection on GitHub:
https://github.com/open-cluster-management/policy-collection
The stable folder has the example policies included in the product, and the community folder has community developed policies that may eventually be adopted officially in the stable folder.
