Deploying and Managing Multicloud Applications with IBM Multicloud Manager and Cloud Event Management

Published in

IBM Cloud

13 min readOct 18, 2019

This blog will cover the basics of deploying and managing an application across multiple clouds using IBM Multicloud Manager and the embedded Cloud Event Management capability.

For the purposes of this guide, we will be working with the Guestbook application here.

Prerequisites: You must install and configure the following tools and platforms:

1. Cloudctl
2. Kubectl
3. Helm
4. Git
5. IBM Multicloud Manager hub cluster
6. At least 1 targeted managed cluster connected to your IBM Multicloud Manager hub.

Multicloud Applications Explained

In layman's terms, an Application is a method to relate multiple helm charts as a single resource which can be created, destroyed, and transferred with ease.

Additional capabilities are provided when you implement an application and its custom IBM Multicloud Manager YAML resources. These capabilities include high-level visualizations, extracting configurations, monitoring resources, and more.

To learn more about the custom IBM Multicloud Manager YAML resources, see IBM Multicloud application resource overview. I will share a brief explanation and examples of the resources used in this article.

Application resource

The Application resource creates a high-level view of the application. View the following Application resource example:

apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
  name: {{ template "guestbookapplication.fullname" . }}
  labels:
    app: {{ template "guestbookapplication.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    name: {{ template "guestbookapplication.fullname" . }}
spec:
  selector:
    matchExpressions:
    - key: app
      operator: In
      values:
      - guestbook
      - gbf
      - gbrm
      - gbrs
  componentKinds:
  - group: core
    kind: Pods

Deployable resource

The Deployable resource deploys helm charts. You will need a Deployable resource for each helm chart managed by the application. View the following Deployable resource example:

apiVersion: mcm.ibm.com/v1alpha1
kind: Deployable
metadata:
  name: {{ template "guestbookapplication.fullname" . }}
  labels:
    app: {{ template "guestbookapplication.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    name: {{ template "guestbookapplication.fullname" . }}
    servicekind: ApplicationService
spec:
  deployer:
    kind: helm
    helm:
      chartURL: https://raw.githubusercontent.com/abdasgupta/helm-repo/master/3.2-mcm-guestbook/gbf-0.1.0.tgz
      namespace: {{ .Values.appInClusterNamespace }}

PlacementPolicy resource

The PlacementPolicy resource manages the capability of IBM Multicloud Manager, to find clusters in order to place or deploy workloads.

When you are adding a cluster, removing a cluster, or implementing changes to cluster compliance, IBM Multicloud Manager will regenerate a list of applicable clusters based on your changes. View the following PlacementPolicy resource example:

apiVersion: mcm.ibm.com/v1alpha1
kind: PlacementPolicy
metadata:
  name: {{ template "guestbookapplication.fullname" . }}
  labels:
    app: {{ template "guestbookapplication.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    name: {{ template "guestbookapplication.fullname" . }}
    servicekind: CacheService
spec:
  clusterReplicas: {{ .Values.replicaCount }}
  clusterLabels:
    matchLabels:
{{ toYaml .Values.targetCluster.labelSelector.matchLabels | indent 6 }}
  resourceHint:
{{ toYaml .Values.targetCluster.resourceSelector | indent 4 }}
  compliances: {{ .Values.targetCluster.compliances }}

ApplicationRelationship resource

The ApplicationRelationship resource provides a method to link applications together based on their source and destination values. View the following ApplicationRelationship resource example:

apiVersion: mcm.ibm.com/v1alpha1
kind: ApplicationRelationship
metadata:
  name: {{ template "guestbookapplication.fullname" . }}-appfrontend
  labels:
    app: {{ template "guestbookapplication.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  destination:
    kind: Deployable
    name: {{ template "guestbookapplication.fullname" . }}
  source:
    kind: Application
    name: {{ template "guestbookapplication.fullname" . }}
  type: contains

PlacementBinding resource

The PlacementBinding resource provides the capability to bind a placement policy to a deployable or a set of deployables. View the following PlacementBinding resource example:

apiVersion: mcm.ibm.com/v1alpha1
kind: PlacementBinding
metadata:
  name: {{ template "guestbookapplication.fullname" . }}
  labels:
    app: {{ template "guestbookapplication.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    name: {{ template "guestbookapplication.fullname" . }}
    servicekind: CacheService
placementRef:
  apiGroup: mcm.ibm.com
  kind: PlacementPolicy
  name: {{ template "guestbookapplication.fullname" . }}
subjects:
- apiGroup: mcm.ibm.com
  kind: Deployable
  name: {{ template "guestbookapplication.fullname" . }}

Note: You can specify multiple deployables to be managed by your placement policy. Add other Deployable resources in the subjects field of your resource.

Guestbook Application Introduction

The Guestbook application is composed of 3 services. Each service is comprised of a Deployable, PlacementPolicy, and ApplicationRelationship resources. The application consists of the following components:

Frontend- This is the user interface of the application will communicate with both the redis master and the redis slave instances. With the frontend, you can submit names, which will be stored into the redis master.
Redis master- This is our master database, which will hold the values sent from the frontend. The data sent to the master will be replicated onto the redis slave.
Redis slave- The redis slave will persist the data sent from the master, which allow for enhanced reliability.

View the diagram for the topology of the Guestbook application:

Adding image policies onto all clusters

Create and apply a cluster image policy to access Docker images from specific registries. A cluster image policy can exist at the namespace and cluster level. When you implement the cluster image policy at the namespace level, the namespace has access to the associated registries. When you implement the cluster image policy at the cluster level, every namespace on the cluster has access to the associated registries.

A cluster image policy must be applied to each cluster that will run on the application. Complete the following steps to create a cluster image policy for the Guestbook application:

Create a cluster image policy (cluster-image-policy.yaml) for the Guestbook application. Your resource YAML might resemble the following file:

kind: ClusterImagePolicy
metadata:
  name: gcr-guestbook-registry
spec:
  repositories:
  - name: 'gcr.io/google_samples/gb-redisslave'
    policy:
      va:
        enabled: false
  - name: 'gcr.io/kubernetes-e2e-test-images/redis'
    policy:
      va:
        enabled: false
  - name: 'gcr.io/google-samples/gb-frontend'
    policy:
      va:
        enabled: false

2. Log in to your cluster and apply the gcr-guestbook-registry policy created above onto your target cluster by running the following commands:

$ cloudctl login -a https://<CLUSTER_IP_ADDRESS>:8443 --skip-ssl-validation
$ cloudctl target -n <MULTICLUSTER-NAMESPACE>
$ kubectl apply -f cluster-image-policy.yaml

3. Verify that your cluster image policy was added by running the following command:

$ kubectl get clusterimagepolicy
NAME                                    AGE
gcr-guestbook-registry                  16s
ibmcloud-default-cluster-image-policy   10d

Your gcr-guestbook-registry cluster image policy is added.

Deploying the Guestbook application

The process to deploy a multicloud application is similar to deploying helm charts. Complete the following steps to deploy the application:

Clone the application project environment by running the following command:

$ git clone git@github.ibm.com:IBMPrivateCloud/Think2019.git

2. Archive your application into your cluster by running the following command:

$ helm package guestbook

3. Log in to your cluster and load the application by running the following commands:

$ cloudctl login -a https://<CLUSTER_IP_ADDRESS>:8443 --skip-ssl-validation
$ cloudctl catalog load-chart - archive guestbook-0.1.0.tgz

4. Install the application onto your IBM Multicloud Manager hub cluster by running the following command:

$ helm install guestbook -n <release-name> -tls

The Guestbook application is installed and ready for deployment.

Viewing the application

You can view your application from the command line interface (CLI) by running the following command:

$ kubectl get application

View your application from the IBM Multicloud Manager console.

In the IBM Multicloud Manager Hub side navigation bar, click Applications. Your Applications page might resemble the following diagram:

Updating the application cluster deployments

In order to update and change the cluster(s), where the application is deployed, you must update the placement policies on the application. You can update the cluster deployments from the CLI or IBM Multicloud Manager console.

Complete the following steps to update your cluster deployments from the CLI

Get your placement policies by running the following command, and update the matchLabels object by editing the guestbook-gbapp placement policy in our application:

$ kubectl -n kube-system get placementpolicy
NAME                          REPLICAS   ORDERBY   DECISIONS       AGE
guestbook-gbapp               1          cpu()     local-cluster   7d
guestbook-gbapp-redismaster   1          cpu()     local-cluster   7d$ kubectl -n kube-system edit placementpolicy guestbook-gbapp

You can add and remove cluster labels, in order to match cluster(s) to your application in the matchLabels object. Your updated placement policy might resemble the following section in your file:

spec:
  clusterLabels:
    matchLabels:
      environment: Dev
      name: local-cluster

Note: If you know the field you’d like to edit ahead of time, run the following command from the CLI:

$ kubectl -n kube-system patch placementpolicy guestbook-gbapp -p ‘{“spec”:{“clusterLabels”:{“matchLabels”:{“name”:”factual-beetle”}}}}’placementpolicy.mcm.ibm.com/guestbook-gbapp patched

Complete the following steps to update your cluster deployments from the IBM Multicloud Manager console:

Log in to your IBM Multicloud Manager cluster.
From the navigation menu, click Applications.
Select the guestbook-gbapp application and view the Placement Policies section.
Edit guestbook-gbapp placement policy. Select the Options icon and click Edit. A YAML resource editor appears in the Edit Placement Policy dialog box.

Note: You can add and remove cluster labels in order to match cluster(s) to our application. Update the name parameter in the matchLabels parameter to switch the cluster deployment.

5. Click Submit.

Uninstalling the application — Optional

You can uninstall your application from the CLI or IBM Multicloud Manager console. Note that the following methods only uninstall the chart, but the chart remains in your internal cluster registry.

Uninstall the application and its associated resources by running the following command from the CLI:

$ helm delete <release-name> -purge --tls

Complete the following steps to uninstall the guestbook-gbapp application from the IBM Multicloud Manager console:

Log in to your IBM Multicloud Manager cluster.
From the navigation menu, click Applications.
Click the Options icon and select Remove Application from the guestbook-gbapp row. The Remove Application dialog box appears.
Select the associated resources that you want to delete and click Remove Application.

The guestbook-gbapp application is removed.

Handling Events and Incidents for Application Availability

You can leverage Cloud Event Management to send and respond to events to help monitor and support your application and overall cluster health. By creating and using an API Key, you can send events to your cluster from any source that is able to make a web request. An API key can be created for some or all of the APIs within Cloud Event Management. These API types include — Events API, Incidents API, Users API, Groups API, and Runbooks API.

Cloud Event Management can be leveraged by teams in order to manage incidents from applications and services, including the events generated from them. These events can include performance, compliance, failures, and more. In Cloud Event Management, you can create both incident policies and event policies. Since Cloud Event Management aggregates the incidents and events in a centralized location, we are able to simply perform automated actions (Runbooks), manual actions, or a composition of both to remediate incidents. You can further customize your actions to automatically assign and notify users or groups, as well as managing priorities.

Creating an API Key

Create an API Key from the IBM Multicloud Manager console to send events to your cluster. Complete the following steps:

Log in to your IBM Multicloud Manager cluster.
From the navigation menu, select Event Management.
Select the Administration tab to view your current configurations.
Click API Keys.

5. From the API Keys page, click New API Key +. Before the API Key is created, you will have the option to select which Cloud Event Management resources the API Key will have access to. A description must be filled in, before the API key can be generated.

6. Click Generate.

7. After you click Generate, a Create API key dialog box appears with the following information: API Key Name and the API Key Password.

It is worth taking note of the API Key Password and the permissions selected in the previous screen, as there is no way to retrieve this after this menu has closed.

Now that we have created our API Key we can begin sending events to Cloud Event Management.

Sending events to Cloud Event Management

To make the call to your created API key, you need to first get the Base API URL. From the API Keys page, select the Manage API Keys tooltip. This will open a menu containing the base API URL.

Configuring an Integration

Cloud Event Management allows us to configure integrations to receive incoming events, or send outgoing notifications. In the following example, I will be configuring an outgoing Slack integration to monitor cluster health, allowing me to send events from our cluster to a Slack channel.

Complete the following steps to configure an integration:

Log in to your IBM Multicloud Manager cluster.
From the navigation menu, click Event Management to access Cloud Event Management.
From the Administration page within Cloud Event Management, click Integrations to view your current configured integrations.
To begin creating an outgoing integration, select the Outgoing tab, and click New Integration.

5. Select Configure for the Slack integration to begin creating the Slack integration. For this integration, you will need a name and the Slack web-hook URL. Fill in the corresponding information, ensure the integration is enabled, and click Save. This integration will now show up in your list of outgoing integrations, and can be used throughout Cloud Event Management.

Let’s learn about how you can forward cluster health events to your Slack Channel.

Responding to Incidents with Cloud Event Management

To respond to an incident within Cloud Event Management, you must create an Incident Policy. An incident policy consists three parts: the details, incidents, and actions. You can specify an incident policy to alert you when certain conditions or events are met. For the purposes of this guide, let’s create an incident policy that will send a notification to a slack group (using the integration configured previously mentioned).

Tip: When you configure a new incident policy, it is helpful to first ensure the policy is working with all events before you implement conditions to respond to a subset of events.

Creating an incident policy

Create an incident policy with Cloud Event Management by completing the following steps:

From the Cloud Event Management Administration page, click Policies > Incident Policies. A table appears with a list of your currently configured incident policies. You can view general information about each policy. You can also enable/disable, edit, and delete incident policies.
Click New Incident Policy to configure your incident policy.
Enter the policy name and description. When you configure an incident policy, you can base it off the attributes that the event has, as well as the events that the incident contains.

For this guide, view the Cloud Event Management console screenshots to create an incident policy that will respond to priority 1 events, with a summary containing the string “Cluster Health”.

Incident Attributes

Event Attributes

Tip: Test your incident policy conditions, by clicking Test. This will show you which incidents in the past X days match the specified conditions.

4. Select the appropriate actions from the Action section. For this guide, select the Assign and notify check box to send a notification to Slack.

5. Click Add assignment / notifications. The Select assignees and recipients popup box appears.

6. Select Groups, Users, or Integrations that you want to notify. For this guide, click the Integrations tab and check the Notify check box.

7. Select Apply. You can also set the priority of the event, and enable it when you are ready.

8. To finish creating the event, click Save. When you enable the integration, the incident will begin responding to events.

Now, when a Priority 1 event regarding Cluster Health comes into your Cloud Event Management hub, it will forward a notification to Slack. A user or team can respond to the Slack notification dynamically. You also always have the option to assign a user or group directly if necessary.

Viewing incidents

You can view current incidents by navigating to the Incidents tab from the IBM Multicloud Manager console. Click All Incidents to view a high-level view of your recent incidents.

You can view more details about the incident by clicking Investigate. You will be able to view the Resolution View, Events, and Timeline.

The Resolution View tab contains the following high-level details: the resources affected, suggested runbooks, a snippet of the timeline, and the ability to mark the incident as resolved or in progress.

The Events tab contains a list of all the events which are a part of the incident. To view more details about an event, select the drop-down icon. You can also select the event to view the following details: severity, resource type, first occurrence, summary, and event type.

Notice that Cluster Health is in the Summary column, and that this incident is priority 1. For these reasons, you can expect that your previously configured incident policy sent a notification to our Slack Channel to update the team on this high priority incident.

Your slack message might resemble the following message:

The team has the following response options to the Slack message:

Select Incident List to view all incidents.
Select Incident Viewer to view the specified incident in a detailed view.
Select Ownership to take ownership of this incident as either a user or a group.

Continue to manage your incidents and events with IBM Multicloud Manager Event Manager integration.