Open in app

Sign in

Write

Sign in

Running your management plane with OKE, Verrazzano and Cluster API — Part 2

Ali Mukadam
Verrazzano
Published in
9 min readAug 3, 2023

--

In Part 1, we looked at the different Verrazzano profiles, provisioned an admin cluster and then we used Cluster API (CAPI) to create a workload OKE cluster. We automated the last part by using ArgoCD.

In this article, we will install Verrazzano in the OKE workload cluster. Preferably, we want to:

  • keep things as touchless an approach as possible
  • stick with GitOps and ArgoCD
  • install a Verrazzano with a managed-cluster profile in the workload cluster

Installing Verrazzano is easy:

  1. Install Verrazzano Platform Operator (VPO)
  2. Install a Verrazzano Instance with your required configuration ( which components you want/do not want etc)

The 2nd step above is actually performed by the (VPO). You just tell it by way of a Verrazzano Custom Resource which profile you want and which component you want enabled or disabled. If you are deploying Verrazzano in multi-cluster configuration, there’s 1 additional step you must do after Verrazzano deployment has completed. You must register the managed cluster(s) with the Admin server.

This is easy when you have access to a terminal, kubectl and kubeconfig. You can even script this with Terraform as we’ve done in the Terraform Verrazzano module where we dynamically look up the relevant subnet, and NSG ids and such.

But how to automate all this with CAPI and ArgoCD? Well, CAPI has an experimental Add-on Provider which allows you to install, configure, upgrade and delete Cluster Add-ons and one of these is for Helm, the more or less default Kubernetes package manager. So, we’ll use this addon provider to handle the installation of the Verrazzano Platform Operator for us.

Creating a helm chart for the Verrazzano Platform Operator

The experimental helm provider requires CAPI 1.5. Let’s install this locally where we can interact with our Admin/Management cluster:

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.5.0/clusterctl-linux-amd64 -o clusterctl
chmod +x clusterctl
sudo mv clusterctl /usr/local/bin

Next, let’s initialize the helm provider:

clusterctl init --addon helm -n verrazzano-capi

We need to get a helm chart for the VPO. This is not available yet but something that we’ll hopefully fix soon. In the meantime, let’s create our own and host it in a simple helm repo on GitHub. For the purpose of this article, we’ll use GitHub pages but you can host it in OCI Artifact Repository if you want. Follow this simple guide to create a public helm repo. Then, clone the git repo locally. We are now ready to create a helm chart for the VPO. Run the following command against the Admin server:

kubectl cp -n verrazzano-install $(kubectl get pod -n verrazzano-install -l app=verrazzano-platform-operator -o custom-columns=:metadata.name --no-headers):platform-operator/helm_config/charts/verrazzano-platform-operator -c verrazzano-platform-operator charts/verrazzano-platform-operator

This will extract the CRDs, cluster roles and bindings, deployment etc required to run the operator. Once extracted, package it as a helm chart:

helm package charts/verrazzano-platform-operator
mv verrazzano-platform-operator-1.6.1.tgz ../charts/

Next, we create the index file:

helm repo index --url https://<github pages url>/ .

Check your index.yaml:

apiVersion: v1
entries:
  verrazzano-platform-operator:
  - apiVersion: v2
    appVersion: 1.6.1
    created: "2023-08-02T13:11:49.517054801+10:00"
    description: A Helm chart for the Verrazzano Platform Operator
    digest: 9a3324d7404e1f9c122814369ae8acc89cb143c1861399b90cf4e84a81dee8e0
    name: verrazzano-platform-operator
    type: application
    urls:
    - https://<mygithubid>.github.io/<mygithubrepo>/charts/verrazzano-platform-operator-1.6.1.tgz
    version: 1.6.1
generated: "2023-08-02T13:11:49.514715928+10:00"

Add the compressed chart and index.yaml to the GitHub repo, commit and push. Since you configured GitHub Pages, this will trigger a GitHub action. Wait for it to complete and you can test locally.

First, add the repo:

h repo add vzrepo https://<mygithubid>.github.io/<mygithubrepo>/
h repo update

Next, check if you can see the helm chart:

helm search repo
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
vzrepo/verrazzano-platform-operator     1.6.1           1.6.1           A Helm chart for the Verrazzano Platform Operator

We now have the VPO operator available as a helm chart which we are going to use in the next section.

Deploying the Verrazzano Operator using Cluster API and ArgoCD

We are now going to deploy the Verrazzano Platform Operator. Recall that we created a cluster before using a manifest that CAPI can use. First, we need to add a label “verrazzanoPlatformOperator” to it and set its value to “enabled”.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: vzoke
    verrazzanoPlatformOperator: enabled
  name: vzoke
  namespace: verrazzano-capi
spec:
  controlPlaneRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: OCIManagedControlPlane
    name: vzoke
    namespace: verrazzano-capi
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: OCIManagedCluster
    name: vzoke
    namespace: verrazzano-capi

Next, we’ll create an add-on for helm:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: verrazzano-platform-operator
  namespace: verrazzano-capi
spec:
  clusterSelector:
    matchLabels:
      verrazzanoPlatformOperator: enabled
  repoURL: https://<mygithubid>.github.io/<mygithubrepo>/charts
  chartName: verrazzano-platform-operator
  version: v1.6.1
  releaseName: verrazzano-platform-operator

Check this in the same repo where you deployed the code for the workload cluster. On ArgoCD, you should see the following:

You can use clusterctl or kubectl to obtain the kubeconfig of the OKE Workload Cluster:

clusterctl get kubeconfig vzoke -n verrazzano-capi > vzoke.kubeconfig

If your loyalty is towards kubectl:

kubectl -n verrazzano-capi get secret vzoke-kubeconfig -o json | jq -r .data.value | base64 -d > vzoke.kubeconfig

Check if the VPO has successfully installed:

kubectl --kubeconfig vzoke.kubeconfig -n verrazzano-install get pods
NAME                                                    READY   STATUS    RESTARTS       AGE
verrazzano-platform-operator-7bf5d54d97-2nwr8           1/1     Running   1 (2d1h ago)   2d2h
verrazzano-platform-operator-webhook-7b7c9b8867-hvfl7   1/1     Running   0              2d2h

Similarly, we can check if the CRDs have been created:

kubectl --kubeconfig vzoke.kubeconfig api-resources | grep verrazzano
metricsbindings                                                     app.verrazzano.io/v1alpha1             true         MetricsBinding
metricstemplates                                                    app.verrazzano.io/v1alpha1             true         MetricsTemplate
multiclusterapplicationconfigurations        mcappconf,mcappconfs   clusters.verrazzano.io/v1alpha1        true         MultiClusterApplicationConfiguration
multiclustercomponents                       mccomp,mccomps         clusters.verrazzano.io/v1alpha1        true         MultiClusterComponent
multiclusterconfigmaps                       mccm,mccms             clusters.verrazzano.io/v1alpha1        true         MultiClusterConfigMap
multiclustersecrets                          mcsecret,mcsecrets     clusters.verrazzano.io/v1alpha1        true         MultiClusterSecret
verrazzanomanagedclusters                    vmc,vmcs               clusters.verrazzano.io/v1alpha1        true         VerrazzanoManagedCluster
verrazzanoprojects                           vp,vps                 clusters.verrazzano.io/v1alpha1        true         VerrazzanoProject
verrazzanos                                  vz,vzs                 install.verrazzano.io/v1beta1          true         Verrazzano
ingresstraits                                                       oam.verrazzano.io/v1alpha1             true         IngressTrait
loggingtraits                                                       oam.verrazzano.io/v1alpha1             true         LoggingTrait
metricstraits                                                       oam.verrazzano.io/v1alpha1             true         MetricsTrait
verrazzanocoherenceworkloads                                        oam.verrazzano.io/v1alpha1             true         VerrazzanoCoherenceWorkload
verrazzanohelidonworkloads                                          oam.verrazzano.io/v1alpha1             true         VerrazzanoHelidonWorkload
verrazzanoweblogicworkloads                                         oam.verrazzano.io/v1alpha1             true         VerrazzanoWebLogicWorkload

Now, that the VPO is installed in OKE workload cluster, we can instruct it to deploy Verrazzano.

Creating a helm chart for Verrazzano managed-cluster

In my previous article, I mentioned that we have several profiles and one of these is a managed-cluster profile. The managed-cluster profile’s purpose is to run your workloads. It offloads most of the operational stuff to the admin cluster. You can of course stick to the default and just deploy Verrazzano. However, when deploying with OKE and depending on your use case, there’s an amazing number of configuration options e.g. load balancer subnet, NSG etc. Here, we want to configure 2 things:

  1. the subnet OCID (for a future use case)
  2. the load balancer NSG

For the purpose of this article, we’ll still use the public subnet and NSG. We’ll just use helm to set the values. Similar to the operator, we don’t have a helm chart published yet so we’ll create our own.

helm create verrazzano-instance

Remove all the default manifests under the templates directory. Create a new template verrazzano.yaml. For the purpose of this exercise, we’ve turned off almost everything:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vzoke
spec:
  profile: managed-cluster
  environmentName: vzoke
  components:
    argoCD:
      enabled: false
    clusterAPI:
      enabled: false
    clusterOperator:
      overrides:
      - values:
          syncClusters:
            enabled: true
            clusterSelector:
              matchExpressions:
              - key: vz
                operator: In
                values: [oci]
    coherenceOperator:
      enabled: false
    console:
      enabled: false
    dns:
      wildcard:
        domain: nip.io
    fluentd:
      enabled: true
    grafana:
      enabled: false
    ingressNGINX:
      type: LoadBalancer
      overrides:
      - values:
          controller:
            service:
              annotations:
                service.beta.kubernetes.io/oci-load-balancer-internal: "false"
                service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "All"
                service.beta.kubernetes.io/oci-load-balancer-subnet1: {{ .Values.pub_subnet_id }}
                oci.oraclecloud.com/oci-network-security-groups: {{ .Values.pub_lb_nsg_id }}
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            values:
              global:
                meshID: yggdrasil
                multiCluster:
                  clusterName: vzoke
                network: vzoke
            components:
              egressGateways:
                - name: istio-egressgateway
                  enabled: true
              ingressGateways:
                - name: istio-ingressgateway
                  enabled: true
                  k8s:
                    serviceAnnotations:
                      service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "All"
                      oci.oraclecloud.com/oci-network-security-groups:  {{ .Values.pub_lb_nsg_id }}
    jaegerOperator:
      enabled: false
    kiali:
      enabled: false
    kubeStateMetrics:
      enabled: false
    opensearch:
      enabled: false
    opensearchDashboards:
      enabled: false
    prometheus:
      enabled: false
    prometheusOperator:
      enabled: false
    thanos:
      enabled: false
    rancher:
      enabled: false
    velero:
      enabled: false
    weblogicOperator:
      enabled: false

Notice the template snippets under service annotations. Change the Chart.yaml as well:

apiVersion: v1
appVersion: 1.6.1
description: A Helm chart for the Verrazzano Instance
name: verrazzano-instance
version: 1.6.1

As before, package the chart, move it to the charts directory and update the index:

helm package charts/verrazzano-instance
mv verrazzano-platform-instance-1.6.1.tgz ../charts/
helm repo index --url https://<github pages url>/ .

Add the new chart, index.yaml to the git repo, stage, commit and push to GitHub. Wait for the GitHub Action to finish and then do a search:

helm search repo
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
vzrepo/verrazzano-instance              1.6.1           1.6.1           A Helm chart for the Verrazzano Instance
vzrepo/verrazzano-platform-operator     1.6.1           1.6.1           A Helm chart for the Verrazzano Platform Operator

We can now deploy our Verrazzano instance.

Deploying the Verrazzano managed-cluster profile using Cluster API and ArgoCD

Create a new helm add-on in the repo where you define your cluster:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: verrazzano-instance
  namespace: verrazzano-capi
spec:
  clusterSelector:
    matchLabels:
      verrazzanoPlatformOperator: enabled
  repoURL: https://<mygithubid>.github.io/<mygithubrepo>/charts
  chartName: verrazzano-instance
  version: v1.6.1
  releaseName: verrazzano-instance
  valuesTemplate: |
    pub_subnet_id: {{ index .InfraCluster.spec.networkSpec.vcn.subnets 1 "id" }}
    pub_lb_nsg_id: {{ index .InfraCluster.spec.networkSpec.vcn.networkSecurityGroup.list 2 "id" }}

Stage, commit and push to your Git repo. In ArgoCD, you should now see an additional helm addon created:

After a few mins, you can check Verrazzano’s status:

vz --kubeconfig vzoke.kubeconfig status

Verrazzano Status
  Name: vzoke
  Namespace: default
  Profile: managed-cluster
  Version: 1.6.1
  State: Ready
  Available Components: 11/11

Registering the managed cluster with the Admin Cluster

Finally, we want to register the managed OKE cluster with the Admin cluster. You can use the VMC method or if you’re in a hurry, use the Rancher method. Login to Rancher console and click on “Import Existing”:

Select Generic:

Give it a name and under label, add the following key-value pair:

Then, click Create. You’ll see the following screen:

Since we were in a hurry and didn’t set up certificates as we should have in a proper environment, we’ll use the insecure option just to complete the process to see if things work. Click on the 2nd link and then apply it against the OKE cluster:

curl --insecure -sfL https://rancher.tr-admin.192.9.176.244.nip.io/v3/import/7zxdmnkgnsnt9glxspdkqsf58jhwmz45bxvqtx6mkjjrfbxjflrtjg_c-m-qnl9l9b6.yaml | kubectl --kubeconfig vzoke.kubeconfig apply -f -

After a few mins, you’ll be able to see the cluster as active:

You can now use Rancher to explore your managed cluster.

Summary

In this article, we used CAPI and a CAPI helm addon to install the Verrazzano Operator and a Verrazzano managed-cluster profile on an OKE workload cluster also provisioned by CAPI. We went through a few extra details (such as creating the helm charts) so you can customize your chart as you need. Creating the helm charts and repo is a one-time activity and once you have them, you can repeat the workload cluster creation with OKE, Verrazzano Operator and instance installation and expect it to be “touchless” until the registration point as a managed cluster.

As before, I would like to thank my colleagues Abhishek Mitra and Shyam Radakrishnan for their infectious enthusiasm about Cluster API and their contributions to this article. Stay tuned for Part 3.

Oracle Cloud
Kubernetes
Verrazzano
Cluster Api
Argo Cd

--

--

Ali Mukadam
Verrazzano

Written by Ali Mukadam

477 Followers
Editor for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams