Red Hat ACM’s ManagedCluster Lifecycle in “action”
Red Hat’s Advanced Cluster Management (ACM) gives user’s the ability to manage cloud-based services and data on Kubernetes clusters across multiple cloud providers from a single control plane. ACM leverages the Kubernetes API to deploy its own custom resources and provide additional management capabilities within “imported” clusters. The “managedCluster” resource allows for greater visibility of cost and performance while giving users a new tool to leverage against their own hybrid environments, from the context of a “hub” cluster.
In the blog I will cover the lifecycle of a managedCluster, namely this is a review of the import, update and deletion of a managedCluster system. This review covers ACM’s 2.0 release, and may not be applicable to future releases.
This blog is intended to be of value to engineers who are interested in multicloud management solutions for Kubernetes’ clusters (for industry or otherwise), are already leveraging Red Hat’s multicloud management solution against their own cloud environments, or are planning to contribute to this technology; much of which is already open sourced.
As an illustrative tool, I have segmented the lifecycle into a three-act structure, beginning with import; though there are many scenarios this particular sequence of use (i.e. import, followed by general usage of managedCluster including update, and finally deletion of managedCluster) may not apply to your own usage.
Terminology/Component Glossary:
- hub cluster: A Kubernetes, OpenShift Container Platform (OCP), cluster with Advanced Cluster Management installed, capable of managing subordinate clusters using its managedCluster custom api resource.
Edit: A hub can only be created with an OCP cluster.
- imported cluster: A Kubernetes cluster that has installed klusterlet and klusterlet-addon resources, and is working as a subordinate to a hub cluster. Klusterlet and klusterlet-addons are can be installed on AKS, EKS, GKE, IKS and OCP clusters.
Edit: Thus, AKS, EKS, GKE, IKS and OCP clusters are all capable of being managed by an OCP hub cluster.
- managedCluster: A custom Kubernetes api resource used to represent the status of an imported cluster from the context of a hub cluster.
- managedCluster-import-controller: The controller on the hub cluster responsible for coordinating import on the hub cluster, as well as for reconciling over the state of the managedClusters.
- klusterlet: The controller on the imported cluster responsible for launching and updating klusterlet-registration-controller and the klusterlet-work-agent.
- klusterlet-addon-operator: The controller on the imported cluster responsible for deploying and reconciling the state of klusterlet-addons.
- klusterlet-work-agent (work-agent): The controller on the imported cluster responsible for reconciling manifestWork, and updating the klusterlet.
- klusterlet-addon-controller (registration-controller): The controller on the hub responsible for creating and reconciling the certificate signing request between the the hub and imported cluster.
- manifestWork: An api resource for defining and executing jobs on a cluster.
- finalizer: A label added to prevent the deletion of a resource; it is used to coordinate graceful tear-downs of cluster applications.
Import
- Act one: User Initializes Import
The user initializes the import of a managed cluster from the hub cluster’s ACM console. *(see image below)
From the console of a hub cluster, users have the ability to monitor their imported clusters, import already existing clusters to their hub or create a cluster and import it automatically.
2. Console begins work
On the hub cluster, after “Generate command” is pressed, the UI will create the cluster namespace and managedCluster resource, both are assigned the same name provided by the user for the cluster.* The UI will also create the “klusterletAddonConfig” (the CRD for the klusterlet-addon resource).
The UI then adds a finalizer to the managedCluster resource spec and assigns its key, HubAcceptsClient, a value of “true”, signifying that the hub cluster accepts the join of a new managedCluster.* It also creates a “bootstrap” service account that will allow for the initial registration between the hub and and it’s new managedCluster.
The managedCluster-import-controller then creates the import manifest secret.*
3. User applies the import manifest
The content of the newly created secret is a base64 encoded custom resource definition for klusterlet. The console presents the encoded definition to the user within a kubectl apply command.* The user will take the command and apply it on their soon-to-be imported cluster. It will trigger the creation of the klusterlet on that managedCluster.
4. Enter: The klusterlet
On the managed cluster, klusterlet will rollout the deployment of the registration-controller, followed by the work-agent.
The registration-controller creates a certificate signing request which is shortly thereafter approved by the hub cluster’s managedCluster-import-controller.
The managedCluster-import-controller will then create manifestWork sets to deploy klusterlet-addon-controller. When the controller is up and running, it will then create its own manifestWorks (managed by an addon-work-manger) to launch addon applications for monitoring, search, IAM policy and certificate regulation features for the imported cluster.*
5. End of Act I: work-agent
The work-agent will continue to reconcile manifestWork states over the lifespan of the managedCluster.
At this point the user should have a running managedCluster resource in their hub cluster, and a klusterlet and klusterlet-addon-operator running in their imported cluster. Regular usage of ACM features can begin after the import process has succeeded.
Upgrade
- Act II: User Initializes Upgrade
The user has the option to upgrade a managedCluster’s klusterlet can be initiated by the user from the context of a cluster hub’s web console. *
2. Update to manifest work
Once “Upgrade Cluster” is selected, the managedCluster-import-controller will update the import manifest secret and the manifestWork of the klusterlet.
3. Update Components
The imported cluster’s work-agent will reconcile the changes in manifestWork by beginning work to upgrade the klusterlet. Once complete, the klusterlet will upgrade the registration controller and the work-agent.
Deletion
- Final Act: User Initializes managedCluster deletion
Deletion of a managedCluster will also be user-initiated.*
2. managedCluster-import-controller triggers work-agent reconcile — work-agent deletes klusterlet.
When prompted by the “Destroy” command, the managedCluster-import-controller will delete the manifestWork for the klusterlet. The work-agent’s reconciler will respond to the deletion of ManifestWork by deleting the klusterlet.
3. Final Scene: managedCluster-import-controller deletes ManagedCluster resource and cluster namespace
The managedCluster-import-controller will then wait for the ManagedCluster to go offline — which is indicated to the controller via the managedCluster resource spec, “ManagedClusterConditionAvailable", which represents the availability of the managedCluster.* At this juncture, when the spec is set to “Unknown” or “False” the managedCluster-import-controller will remove all the manifestWork resources in the managedCluster namespace.
Finally, the managedCluster-import-controller’s removes the managedCluster’s finalizer, deletes the managedCluster resource and concludes the deletion process by removing the managedCluster’s namespace. This marks the end of a managedCluster’s lifecycle.
Post Script
For more details on what can be accomplished with an imported cluster, see ACM’s product page or the Open-Cluster-Management organization on Github.
I want to offer a tremendous thank you to Hao Liu, Dominique Vernier, Hanqiu Zhang, and Leena Jawale, Brandi Swope and Mikela Dockery for their input on this blog!