IBM Cloud Pak for Watson AIOps

Lift and shift Watson AIOps with IBM Cloud Pak for Multicloud Management

Authors: Ricardo Olivieri, Radu Mateescu

Ricardo Olivieri
IBM Cloud Pak for AIOps
18 min readJun 2, 2021

--

In this article, we will show how to leverage IBM Cloud Pak for Multicloud Management v2.3 for “lifting and shifting” an IBM Cloud Pak for Watson AIOps v3.1 installation from one OpenShift cluster to another.

Prerequisites

To get the most out of this article, you should have a good understanding of container orchestration platforms and related concepts, specifically Kubernetes and OpenShift. If you are not, there are lots of great online resources that can help you to take advantage of this material (see the Resources section below). You should also be familiar with the installation steps for Watson AIOps v3.1.

Introduction

It is common for many organizations to deploy and maintain applications that spread across on-premises environments and several cloud platforms, such as IBM Cloud, Microsoft’s Azure, or Amazon Web Services (AWS). By leveraging a multi-cloud and hybrid approach, organizations can prevent vendor lock-in and can also take advantage of the offerings and services that make a cloud provider stand out from others. However, the flexibility that such architectures provide do introduce the complexity of managing and maintaining applications and resources that reside in heterogeneous environments. IBM Cloud Pak for Multicloud Management aims to address precisely this type of complexity by providing governance capabilities and visibility of resources across different environments and platforms. For instance, using IBM Cloud Pak for Multicloud Management, developers can see the deployments, pods, and other k8s artifacts, while systems reliability engineers can look at the infrastructure that hosts those artifacts such as clusters and nodes. With IBM Cloud Pak for Multicloud Management, IT organizations have the necessary tools to ensure that an enterprise runs smoothly.

One of the key features of IBM Cloud Pak for Multicloud Management is a consistent mechanism for deploying applications across clusters. In this article, we will show how to leverage this deployment mechanism for “lifting and shifting” an IBM Cloud Pak for Watson AIOps v3.1 installation from one OpenShift cluster to another. The IBM Cloud Pak for Watson AIOps empowers teams to assess, diagnose, and resolve issues quickly and with confidence by leveraging AI technology. With Watson AIOps, teams get real time insights from IT data that otherwise looks dispersed and not related like topology information, logs, observability metrics, resolved incident tickets, and more. Watson AIOps uses predictive modeling to alert ops teams when anomalies are detected before they impact end users. All of this reduces human error and significantly lowers downtime, while at the same time improves the efficiency of IT teams.

Lift and shift Watson AIOps

The high-level steps below outline how IBM Cloud Pak for Multicloud Management can be used to orchestrate the “lifting and shifting” of the IBM Cloud Pak for Watson AIOps:

  1. Watson AIOps is installed on an OpenShift that satisfies its installation pre-requisites (e.g. cluster has the required storage classes; external traffic is allowed to reach the routes; there is enough computational resources; etc.). We use the term source cluster to refer to this cluster. The name for our source cluster is helmet. Note that at this point, IBM Cloud Pak for Multicloud Management is not involved in the installation process of Watson AIOps. Instead, the steps described in the Watson AIOps installation documentation are followed.
  2. A second OpenShift cluster that also satisfies the pre-requisites for hosting Watson AIOps is provisioned. We refer to this second cluster as the target cluster. The name of our target cluster is popcorn.
  3. Both OpenShift clusters, source and target, are imported into IBM Cloud Pak for Multicloud Management.
  4. IBM Cloud Pak for Multicloud Management is used to “discover” Watson AIOps on the source cluster. Here, we validate that IBM Cloud Pak for Multicloud Management can find those top-level Kubernetes artifacts that were provisioned on the source cluster at the time Watson AIOps was installed.
  5. Using those top-level Kubernetes artifacts from the previous step, IBM Cloud Pak for Multicloud Management assembles and creates the necessary assets for provisioning Watson AIOps on the target cluster (i.e. “lift and shift” Watson AIOps from the source cluster to the target cluster).
  6. IBM Cloud Pak for Multicloud Management deploys the assembled application for Watson AIOps to the target cluster.

For this article, we assume steps #1 and #2 have been completed. Therefore, we will focus on steps #3 through #6. Also, you should note that in our source cluster environment, we installed Watson AIOps in the aiops namespace.

Patch required for IBM Cloud Pak for Multicloud Management v2.3

A patch that is not currently available in IBM Cloud Pak for Multicloud Management v2.3 needs to be applied (note that this patch will be included in the subsequent versions of IBM Cloud Pak for Multicloud Management).

1. Log on to the cluster where IBM Cloud Pak for Multicloud Management is installed and run the following command:

2. Locate the assembler element under the resource spec and add the imageand imagePullPolicy attributes as shown below:

3. Save the updates and exit the text editor.

4. Finally, run the following command and make sure all three containers in the cp4mcm-hybridapp pod are running.

Import source and target clusters into IBM Cloud Pak for Multicloud Management

For IBM Cloud Pak for Multicloud Management to lift and shift Watson AIOps, you need to import your source and target clusters as managed clusters into IBM Cloud Pak for Multicloud Management. Let’s take note of the following two terms before proceeding any further:

  • Hub cluster — Refers to the cluster where IBM Cloud Pak for Multicloud Management is installed. You can think of this cluster as the central controller that brings together information from the clusters where your applications run: the hub cluster maintains the state of your applications and clusters where they run.
  • Managed cluster — Refers to a cluster that runs your applications and is connected to the hub cluster. A managed cluster establishes an outbound connection to the hub cluster, receives tasks from it, executes those tasks, and returns the results. In our scenario, the source and target clusters are both managed clusters.

The steps in this section were validated against IBM Cloud Pak for Multicloud Management v2.3 and Red Hat Advanced Cluster Management (ACM) v2.1. If newer versions of ACM are used with IBM Cloud Pak for Multicloud Management v2.3, the Import an existing cluster by running a command launch-out link (as shown in step #2 below) will unfortunately not work. If using a later version of ACM (e.g. v2.2, v2.3) with IBM Cloud Pak for Multicloud Management v2.3, a workaround is to use the Red Hat ACM user interface to generate the import command. Please note this limitation will be addressed in the next release of IBM Cloud Pak for Multicloud Management. Let’s now import the source cluster by following the steps below.

1. Log on to the IBM Cloud Pak for Multicloud Management console and use the navigation menu on the left to access Automate infrastructure -> Clusters.

IBM Cloud Pak for Multicloud Management — Automate infrastructure menu options
IBM Cloud Pak for Multicloud Management — Managed clusters list

2. Click Add a cluster and under Using Red Hat Advanced Cluster Management section, select Import an existing cluster by running a command, and then click Launch.

IBM Cloud Pak for Multicloud Management — Launch-out link to Red Hat ACM

3. On the Import an existing cluster page, enter the cluster name and then click Save and import and generate code.

Red Hat ACM — Import a cluster panel

4. Log on to the source cluster using the OpenShift CLI. Copy the generated command and run it on your source cluster (i.e. the cluster where Watson AIOps was installed).

Red Hat ACM — Import a cluster panel

5. On the source cluster, navigate to the open-cluster-management-agent namespace (i.e. project) and ensure all pods are running:

6. Also on the source cluster, create the Application custom resource definition (you should download its YAML definition file to your local workstation):

7. Log on to the hub cluster using the OpenShift CLI. Using your browser, navigate to Enabling Hybrid Applications when integrated with Red Hat Advanced Cluster Management and follow the instructions for enabling the hybrid application discoverer on your managed cluster. Note that you will need to update the value for thenamespace and cluster-name fields with the name of your source cluster (i.e. look for the string managedcluster and replace it with your cluster name).

8. Back again on the source cluster, you should verify the endpoint-appmgr pod is now running:

The logs for the endpoint-appmgr pod should look similar to this:

9. Back on the IBM Cloud Pak for Multicloud Management console, you should now find an entry for the source cluster (e.g. helmet) that you just imported:

IBM Cloud Pak for Multicloud Management — Managed clusters list

10. Finally, you should now import the target cluster (e.g. popcorn) by following the same steps we took for importing the source cluster. Once the target cluster is imported, you should find an entry for it in the IBM Cloud Pak for Multicloud Management console as well.

IBM Cloud Pak for Multicloud Management — Managed clusters list

Application management

IBM Cloud Pak for Multicloud Management provides application management capabilities for deploying applications and application updates across different and dispersed environments. IBM Cloud Pak for Multicloud Management makes this possible by using a hybrid application model for representing and managing application components. This model includes default Kubernetes artifacts along with custom resource definitions (CRD).

Hybrid application model for Watson AIOps

To define the hybrid application model for Watson AIOps, you should be very familiar with its installation process as outlined in the IBM Documentation. Otherwise, you will not have the necessary knowledge to identify the top-level Kubernetes assets provisioned during the installation phase. Identifying these components is the most critical step to successfully “lift and shift” Watson AIOps from the source cluster to the target cluster (in case you are wondering — by top-level components, we mean those components that do not have owner references — for details, please see Owners and dependents). After reviewing in detail the installation process for Watson AIOps, we identify the artifacts in the tables below as the top-level components that should make up the hybrid application model.

Pre-requisites #1

Pre-requisites #2

AIOps Instance

Event Gateway Instance

Note that we categorized the top-level components of Watson AIOps into deployment phases as shown by each table title above (e.g. Pre-requisites #1, Pre-requisites #2, etc.). Lifting and shifting the different top-level components that make up Watson AIOps must occur in a predefined order to avoid deployment errors. For instance, lifting and shifting the Event Gateway instance before deploying any of the other components will result in failures. One important tip to always keep in mind when defining hybrid application models is that you cannot lift and shift in the same deployment phase an operator subscription and any custom resource instance that results from the installation of that operator. The deployment of a custom resource instance cannot proceed if the custom resource definition does not exist yet on the target cluster (you can think of this scenario as the equivalent to a ClassNotFoundException in Java).

Discovery and assembly of hybrid application for Watson AIOps

1. For IBM Cloud Pak for Multicloud Management to discover the top-level components of Watson AIOps on the source cluster, we need to label them accordingly.

To label such components you can use the OpenShift CLI or the OpenShift console. For instance, if using the CLI, after logging in to your source cluster, you label the ibm-aiops-catalog CatalogSource instance with the following command:

2. Once all top-level components have been properly labeled, it’s time to initiate discovery. Save the Application definition shown below to your workstation in a file named model-pre-req1.yaml. Replace the <source cluster> placeholder with the name of your source cluster.

3. Log on to the hub cluster using the OpenShift CLI and create a new namespace named aiops-3-1. Use the above Application definition, pre-reqs1-aiops, for discovering the top-level artifacts that are part of the Pre-requisites #1 deployment phase:

4. Let’s verify that IBM Cloud Pak for Multicloud Management discovered the right artifacts by executing the following command on the hub cluster (remember to replace the <source cluster> placeholder with the cluster name of your source cluster):

You should see a Deployable.core.hybridapp.io instance for each artifact that makes up the Pre-requisites #1 deployment phase as shown above.

5. Proceed now to uncomment the line tools.hybridapp.io/hybrid-discovery-create-assembler: "true" in the model-pre-req1.yaml file, and then apply the Application definition again:

The above action assembles the hybrid deployables required for lifting and shifting Watson AIOps from the source cluster to a target cluster.

Also, a placement rule is created, which specifies the managed clusters where these components should reside:

6. Inspect the placement rule and look for the targets section. You should see an entry for the source cluster as shown below:

We will now make an amendment to the targets section in the placement rule to add one more target entry. This new target entry will instruct IBM Cloud Pak for Multicloud Management to deploy the components that are part of the Pre-requisites #1 deployment phase to the target cluster for Watson AIOps:

7. After adding the new target entry to the placement rule that points to the cluster where we want to install Watson AIOps, the Pre-requisites #1 deployment phase begins. You should now see Deployable.core.hybridapp.io instances for the target cluster under the corresponding namespace on the hub cluster:

8. You should verify that the above components are successfully deployed by accessing the OpenShift Console of your target cluster:

Target cluster — Installed operators for Pre-requisites #1

9. Once we’ve verified that all artifacts in the Pre-requisites #1 deployment phase are successfully deployed on the target cluster, you can then proceed with the discovery and deployment of the artifacts that belong to the Pre-requisites #2 phase, followed by the AIOps Instance and Event Gateway Instance phases. The steps are identical for each deployment phase except, of course, you use different Applications files for each phase as described next. Remember that you must wait for each deployment phase to complete before moving on to the next one.

For Pre-requisites #2, use the Application definition shown below (save it to a file named model-pre-req2.yaml). As before, replace the <source cluster> placeholder with the name of your source cluster.

For AIOps Instance, use the Application definition shown below (save it to a file named model-instance.yaml). As before, replace the <source cluster> placeholder with the name of your source cluster.

For Event Gateway Instance, use the Application definition shown below (save it to a file named model-em-gateway.yaml). As before, replace the <source cluster> placeholder with the name of your source cluster.

Once you have followed through the above steps for lifting and shifting the components that belong to each one of the deployment phases, you should have a second instance of Watson AIOps running on the target cluster. Using the navigation menu on the OpenShift console of your target cluster, go to Operators, then Installed Operators, and choose the namespace where Watson AIOps was installed on the source cluster. You should see that all required operators for Watson AIOps are also installed on the target cluster as shown in the images below.

Target cluster — Installed operators
Target cluster — Cloud Park for Watson AIOps installation instance
Target cluster — Event Manager gateway instance

Using also the navigation menu on the OpenShift console of your target cluster, you can go to Workloads, then Pods, and choose the namespace where Watson AIOps was installed on the source cluster to see the Watson AIOps pods running.

Target cluster — Watson AIOps pods

Finally, on the IBM Cloud Pak for Multicloud Management console, you can access each one of the hybrid application models for Watson AIOps to get a topology diagram. On the IBM Cloud Pak for Multicloud Management console, use the navigation menu on the left to access Manage Applications -> Hybrid applications. Note that in each topology diagram you can see both, the source cluster (e.g. helmet) and the target cluster (e.g. popcorn).

IBM Cloud Pak for Multicloud Management — Manage applications menu options
IBM Cloud Pak for Multicloud Management — Applications list
IBM Cloud Pak for Multicloud Management — Topology for Pre-requisites #1
IBM Cloud Pak for Multicloud Management — Topology for Pre-requisites #2
IBM Cloud Pak for Multicloud Management — Topology for AIOps Instance
IBM Cloud Pak for Multicloud Management — Topology for Event Gateway Instance

As reference, we show below the hybrid deployables, placement rules, and deployables that you should have on the hub cluster after completing all deployment phases.

Known limitations

It is worth mentioning that the solution described in this article does not handle application state. Application state (i.e. movement of persistent data between clusters) is handled by a different set of tools and falls outside the scope of this article.

Also, if Watson AIOps is uninstalled from a managed cluster either by deleting the hybrid application or by removing a target cluster in a placement rule, custom resources should then be removed by running the uninstall and cleanup script provided in the product documentation.

Conclusion

Once a deployed application is discovered and the hybrid application is created, we can further use that as a deployment blueprint which is decoupled from the source cluster. For example, if Watson AIOps in the source cluster is uninstalled, our discovered hybrid application for Watson AIOps in IBM Cloud Pak for Multicloud Management contains all the information needed to deploy Watson AIOps to any OpenShift cluster. For example, you can use case the lift and shift approach for easily deploying Watson AIOps across your development, staging, and production environments.

Though there is certainly a learning curve for leveraging IBM Cloud Pak for Multicloud Management for lifting and shifting Watson AIOps, it is worth the benefits. Alternatives to the lift and shift approach are implementing an Ansible playbook or shell scripts for deploying Watson AIOps. However, the amount of effort for taking either approach is larger than defining the declarative hybrid application models we covered in this article. Maintenance of the automation logic is also simplified when using the lift and shift method.

Resources

--

--