Develop your own policy controller to integrate with IBM Cloud Pak for Multicloud Management governance

Acknowledgements: Ali Kanso, Jaya Ramanathan, Mikela Dockery and Yanni Zhang

Overview

IBM Cloud Pak for Multicloud Management governance enable enterprises to meet internal standards for software engineering, secure engineering, resiliency, security, and regulatory compliance for workloads hosted on hybrid clouds. To facilitate it, IBM Cloud Pak for Multicloud Management provides the out-of-the-box policy templates that enterprises can use to apply the security check and remediate any violation in your policies.

Furthermore, enterprises can integrate third-party controls with IBM Cloud Pak for Multicloud Management governance by implementing custom policy controllers using the governance policy framework. Check out our related blog, Applying governance capability of IBM Cloud Pak for Multicloud Management for more information.

In this article, I am going to show you the policy framework architecture for IBM Cloud Pak for Multicloud Management governance, how to write a custom controller that does custom security check and deploy it along with IBM Cloud Pak for Multicloud Management governance, and show the result through the governance dashboard.

Prerequisites

• IBM Cloud Pak for Multicloud Management
• Knowledge of Kubernetes and custom resource
• Red Hat operator-sdk

IBM Cloud Pak for Multicloud Management Governance policy framework architecture

IBM Cloud Pak for Multicloud Management Governance policy framework architecture

The design of IBM Cloud Pak for Multicloud Management governance policy framework follows standard Kubernetes development. It consists of following components:

1. Dashboard — IBM Cloud Pak for Multicloud Management Governance and Risk dashboard that allows you to manage policies and view policy results
2. Policy CRD — A Kubernetes Custom Resource Definition (CRD) that acts as a vehicle to pass embedded policy definition from hub to managed clusters
3. Policy framework controller — A Kubernetes controller that propagates policies, and synchronizes status and results between hub and managed clusters
4. Policy controller — A Kubernetes controller that implements the policy against the security or configuration control, monitor the cluster state against the policy definition, report the result and remediate if possible.

Writing a custom policy controller

In order to write a custom policy controller that can be integrated with IBM Cloud Pak for Multicloud Management governance, you need to implement the following flow:

Source: https://static.swimlanes.io/7419f5dc320dc6904c0871cb6fc372c9.png

According to the flow diagram, your policy controller needs to have following capabilities:

1. Watch and retrieve custom policy CR
2. Based on policy retrieved, extract policy definition and call security control APIs to execute the policy
3. Retrieve the execution results and update the corresponding policy
4. Generate an event on a Multicloud Management policy CR with the remediation results

To make it easier, we have open sourced a sample controller, which implements the described flow. View the following GitHub link for more information:

IBM/multicloud-operators-policy-controller

The multicloud-operators-policy-controller repository provides all the code necessary to integrate with IBM Cloud Pak for Multicloud Management governance. It also provides a sample policy to check if your cluster is compliant with the required number of role binding and cluster role binding rules. For more information, please read the adoption guide in the repository.

Create your own policy and controller

It is very easy to customize the sample controller provided in multicloud-operators-policy-controller repository and make it your own.

1. Clone the multicloud-operators-policy-controller repository:

git clone git@github.com:IBM/multicloud-operators-policy-controller.git

2. Customize the policy and controller.

A policy file defines the desired state of the cluster it needs to conform to. The sample policy resembles the following content:

To make your own policy, you must change the policy schema definition. View the following example to change the schema definition. Let’s create a custom policy namedTestPolicy. The schema file is located in the IBM repository.

Your policy might resemble the following content:

metadata: 
  name: testpolicies.policies.ibm.com
spec:
  group: policies.ibm.com
  names:
    kind: TestPolicy
    listKind: TestPolicyList
    plural: testpolicies
    singular: testpolicy

Update the policy controller to watch for TestPolicy kind. Run the following command:

for file in $(find . -name "*.go" -type f); do  sed -i "" "s/SamplePolicy/TestPolicy/g" $file; done
for file in $(find . -name "*.go" -type f); do  sed -i "" "s/samplepolicy-controller/testpolicy-controller/g" $file; done

Now that you updated the controller and policy schema with kind TestPolicy, we can recompile and run the project to see if it really works.

To do that, configure kubectl to IBM Cloud Pak for Multicloud Management cluster by completing the following steps:

• Log in to your cluster.
• Select the user icon, then click Configure client.
• Copy and paste the configuration information into your command line, and press Enter.

Run following commands to install TestPolicy CRD and start the controller:

export GO111MODULE=on
kubectl apply -f deploy/crds/policies.ibm.com_samplepolicies_crd.yaml
operator-sdk run --local --verbose

The output indicating that a controller called testpolicy-controller is running might resemble the following content:

{“level”:”info”,”ts”:1578503280.511274,”logger”:”controller-runtime.manager”,”msg”:”starting metrics server”,”path”:”/metrics”}
{“level”:”info”,”ts”:1578503281.215883,”logger”:”controller-runtime.controller”,”msg”:”Starting Controller”,”controller”:”testpolicy-controller”}
{“level”:”info”,”ts”:1578503281.3203468,”logger”:”controller-runtime.controller”,”msg”:”Starting workers”,”controller”:”testpolicy-controller”,”worker count”:1}
Waiting for policies to be available for processing…

Let’s create a policy and verify that the controller retrieves it and applies the policy onto the cluster. Run following command in the new terminal to apply the policy:

kubectl apply -f deploy/crds/policies.ibm.com_v1alpha1_samplepolicy_cr.yaml

Once the policy is applied, a message appears indicating the policy has been detected and executed by your controller:

{"level":"info","ts":1578503685.643426,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
{"level":"info","ts":1578503685.855259,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
Available policies in namespaces: 
namespace = kube-public; policy = example-samplepolicy 
namespace = default; policy = example-samplepolicy 
namespace = kube-node-lease; policy = example-samplepolicy 

Run the following command to view the policy execution result:

kubectl get testpolicy example-samplepolicy -oyaml

Your output might resemble the following status field:

status:
  compliancyDetails:
    example-samplepolicy:
      cluster-wide:
      - 5 violations detected in namespace `cluster-wide`, there are 0 users violations
        and 5 groups violations
      default:
      - 0 violations detected in namespace `default`, there are 0 users violations
        and 0 groups violations
      kube-node-lease:
      - 0 violations detected in namespace `kube-node-lease`, there are 0 users violations
        and 0 groups violations
      kube-public:
      - 1 violations detected in namespace `kube-public`, there are 0 users violations
        and 1 groups violations
  compliant: NonCompliant

3. Change the policy rules and execution logic.

Let’s update your created policy by changing the policy rules and execution logic. View the following policy rules and execution logic for the sample policy:

maxClusterRoleBindingGroups: 10
maxClusterRoleBindingUsers: 10
maxRoleBindingGroupsPerNamespace: 2
maxRoleBindingUsersPerNamespace: 2

Complete the following steps to customize the policy rules and execution logic:

1. Add new fields in CRD file to introduce new rules. For examples to add a new field see, the IBM repository CRD file.
2. Update the TestPolicySpec structure in samplepolicy_types.go with new fields.
3. Update PeriodicallyExecSamplePolicies function in samplepolicy_controller.go with new logic to execute the policy.
4. Recompile and have fun!

You created a policy controller and policy.

Deploy your controller to the cluster

If you follow the above steps to create your own policy controller, you should now have a custom policy and controller ready to use with IBM Cloud Pak for Multicloud Management governance.

Complete the following steps to build controller image and deploy to your cluster:

1. Run following command to build the controller image:

operator-sdk build <username>/multicloud-operators-policy-controller:latest

2. Push the image to a repository of your choice, e.g. docker hub. Run the following commands:

docker login 
docker push <username>/multicloud-operators-policy-controller

3. Configure kubectl to point to a cluster managed by IBM Cloud Pak for Multicloud Management.

4. Replace the operator manifest to use the built-in image name and update the namespace to watch for policies. The namespace must be the cluster namespace.

sed -i "" 's|ibm/multicloud-operators-policy-controller|ycao/multicloud-operators-policy-controller|g' deploy/operator.yaml
sed -i "" 's|value: default|value: <namespace>|g' deploy/operator.yaml

5. Update the RBAC role by running the following commands:

sed -i "" 's|samplepolicies|testpolicies|g' deploy/cluster_role.yaml
sed -i "" 's|namespace: default|namespace: <namespace>|g' deploy/cluster_role_binding.yaml

6. Deploy the controller to your cluster by running the following commands:

# Setup Service Account
kubectl apply -f deploy/service_account.yaml -n <namespace>
# Setup RBAC for operator
kubectl apply -f deploy/role.yaml -n <namespace>
kubectl apply -f deploy/role_binding.yaml -n <namespace>
# Setup RBAC for SamplePolicyController
kubectl apply -f deploy/cluster_role.yaml
kubectl apply -f deploy/cluster_role_binding.yaml
# Setup the CRD
kubectl apply -f deploy/crds/policies.ibm.com_samplepolicies_crd.yaml
# Deploy the multicloud-operators-policy-controller
kubectl apply -f deploy/operator.yaml -n <namespace>

7. Run following command to verify that the the controller is in running state:

kubectl get pod -n <namespace>

8. Look at the log and make sure everything is running correctly by running the following commands:

Yus-MacBook-Pro:app-operator ycao$ kubectl logs -f -n <namespace> multicloud-operators-policy-controller-854749c8b8-8v9tp
{"level":"info","ts":1578516495.9202724,"logger":"cmd","msg":"Go Version: go1.13.5"}
{"level":"info","ts":1578516495.9204354,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1578516495.9204445,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1578516495.9219334,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1578516498.3849623,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1578516498.4195356,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1578516500.7812169,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1578516500.7819993,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1578516500.7826676,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516500.787235,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516505.7244387,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"multicloud-operators-policy-controller-metrics","Service.Namespace":"default"}
{"level":"info","ts":1578516508.310978,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"multicloud-operators-policy-controller-metrics\" already exists"}
{"level":"info","ts":1578516508.3131533,"logger":"cmd","msg":"Starting the Cmd."}
Waiting for policies to be available for processing...
{"level":"info","ts":1578516508.313996,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1578516508.4145792,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testpolicy-controller"}
{"level":"info","ts":1578516508.5149467,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testpolicy-controller","worker count":1}
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...

Create and manage policy with the IBM Cloud Pak for Multicloud Management dashboard

Now your policy controller has been installed and running on your cluster. Next, create the policy and apply to the cluster using IBM Cloud Pak for Multicloud Management dashboard. Complete the following steps to navigate the console:

1. Log in to IBM Cloud Pak for Multicloud Management console and click on Govern risk from the menu navigation. The Governance and risk dashboard appears.

2. Click the Create policy button. Copy and paste your policy into the YAML editor. The TestPolicy is embedded inside Policy using policy-templates. Your policy might resemble the following content:

3. Select the cluster where your policy controller runs by updating the Cluster binding field. Click the drop-down menu and select.

4. Click Create button. Your created policy appears on the Policies tab.

Wait for a few seconds and let policy framework propagate the policy to your managed cluster and collect the result. Results for your cluster violation appears. It is represented as the following result 1/1.

5. Click on the policy row and drill down into the details of the policy. The Details tab might resemble the following page:

6. Click on Violations tab to view the violation details. Results from the Violations tab might resemble the following page:

That’s it! You have successfully created a custom policy, deployed it to your cluster, and checked the results by using the Governance and risk dashboard.

Conclusion

IBM Cloud Pak for Multicloud Management governance provides an extensible framework for enterprises to introduce their own security policies. By writing a custom policy controller, enterprises can easily integrate existing enterprise security and configuration controls with IBM Cloud Pak for Multicloud Management governance. Enterprises can also take advantage of the Governance and risk dashboard to create and manage both out-of-box policies and custom policies, monitor the policy compliance status, and take further remediation actions.