Deploying a multi-cluster Verrazzano using OKE — Part 2

In the Part 1, we discussed setting up the network infrastructure for a multi-cluster Verrazzano deployment. In this article, we will do the configuration of the clusters so they behave as a kind of global cluster. Below is the multi-clustering process depicted graphically:

Verrazzano multi-cluster deployment and registration process

Recall that a Verrazzano multi-cluster has 1 Admin cluster and 1 or more managed clusters and that each Verrazzano cluster is a Kubernetes cluster:

Verrazzano multi-cluster architecture

Also recall that we have the following setup:

and we chose the Admin server to be in the Singapore OCI region.

We will install Verrazzano with dev/prod profile on the Admin cluster and with managed-cluster profile on the managed clusters.

Note on using kubectx

In the commands below, I use kubectx to set the Kubernetes context where a context is equivalent to a Kubernetes cluster. Strictly speaking that’s not true but it serves our purpose here. Since we have 1 Admin servers and 3 managed servers in 4 different regions, we have 4 different contexts:

Verifying your Kubernetes context

To ensure we are always using the correct context, I execute the kubectx <context-name> before every command.

Installing Verrazzano as Admin

Installing Verrazzano as the Admin cluster is straightforward. You follow the quickstart guide and you can choose between the dev/prod profile. On the operator host, ensure your context is pointing to “admin”:

Verifying your Kubernetes context

If it’s pointing to one of the other clusters, change it as follows:

kubectx admin

We can now begin the installation:

kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.0.3/operator.yaml

Wait for the deployment to finish:

kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator

And confirm the operator pods are working correctly:

[opc@v8o-operator ~]$  kubectl -n verrazzano-install get pods
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-54cf56884f-46zzk   1/1     Running   0          91s

Next, install Verrazzano:

kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: admin
spec:
  profile: dev
EOF

Now we need to wait until the installation is complete:

kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete \
    verrazzano/admin

This will take a while. In the meantime, let’s install Verrazzano on the managed clusters.

Installing Verrazzano on managed clusters

Change the context to 1 of the managed clusters and install the operator again e.g.

kubectx sydney
kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.0.3/operator.yaml
kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator

Repeat the above for all the managed clusters. Before running in each managed cluster, ensure you have changed your context with kubectx <contextname> as above.

Using the same procedure as for the Admin region, verify that the Verrazzano operator has been successfully installed.

Now, install Verrazzano for each using the managed profile by changing the context and name accordingly:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: sydney
spec:
  profile: managed-cluster

Verifying the Admin cluster and managed clusters

While the managed clusters are being installed, let’s see if we can access the various consoles. Ensure you can login into the Verrazzano and Rancher consoles.

Changed the context again and verify:

kubectx sydney
kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete \
    verrazzano/sydney

Repeat the verification for each managed cluster.

Registering the managed clusters

Verify the the CA certificate type for each managed cluster:

kubectx sydney
kubectl -n verrazzano-system get secret system-tls -o jsonpath='{.data.ca\.crt}'

If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority and you can generate a secret containing the CA certificate in YAML format. If it’s not empty, then the certificate is self-signed and needs to be extracted. Refer to the workflow at the beginning of this article.

kubectx sydney
CA_CERT=$(kubectl \
    get secret system-tls \
    -n verrazzano-system \
    -o jsonpath="{.data.ca\.crt}" | base64 --decode)
kubectl create secret generic "ca-secret-sydney" \
  -n verrazzano-mc \
  --from-literal=cacrt="$CA_CERT" \
  --dry-run=client -o yaml > managedsydney.yaml

Repeat the above for the 2 other regions replacing the region/context and filenames accordingly.

Create 3 secrets on the Admin cluster that contains the CA certificate for each managed cluster:

kubectx admin
kubectl apply -f managedsydney.yaml
kubectl apply -f managedmumbai.yaml
kubectl apply -f managedtokyo.yaml

Get the cluster name for the Admin Cluster:

kubectl config get contexts

Cluster names

Get the API Server address for the Admin server:

kubectx admin
export CLUSTER_NAME="cluster-cillzxw34tq"
API_SERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$CLUSTER_NAME\")].cluster.server}")

Create a ConfigMap that contains the Admin cluster’s API server address:

kubectx admin
kubectl apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${API_SERVER}"
EOF

Create the VerrazzanoManagedCluster object for each managed cluster:

kubectx admin
kubectl apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: sydney
  namespace: verrazzano-mc
spec:
  description: "Sydney VerrazzanoManagedCluster object"
  caSecret: ca-secret-sydney
EOF
kubectl apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: mumbai
  namespace: verrazzano-mc
spec:
  description: "Mumbai VerrazzanoManagedCluster object"
  caSecret: ca-secret-mumbai
EOF
kubectl apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: tokyo
  namespace: verrazzano-mc
spec:
  description: "Tokyo VerrazzanoManagedCluster object"
  caSecret: ca-secret-tokyo
EOF

Wait for the VerrazzanoManagedCluster resource to reach the Ready status:

kubectx admin
kubectl wait --for=condition=Ready \
    vmc sydney -n verrazzano-mc
kubectl wait --for=condition=Ready \
    vmc sydney -n verrazzano-mc
kubectl wait --for=condition=Ready \
    vmc sydney -n verrazzano-mc

Export a YAML file created to register the managed cluster:

kubectx admin
kubectl get secret verrazzano-cluster-sydney-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > registersydney.yaml
kubectl get secret verrazzano-cluster-mumbai-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > registermumbai.yaml
kubectl get secret verrazzano-cluster-tokyo-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > registertokyo.yaml

On each managed cluster, apply the registration file:

kubectx sydney
kubectl apply -f registersydney.yaml
kubectx mumbai
kubectl apply -f registermumbai.yaml
kubectx tokyo
kubectl apply -f registertokyo.yaml

Now verify whether the registration completed successfully:

kubectx admin
kubectl get vmc sydney -n verrazzano-mc -o yaml
kubectl get vmc mumbai -n verrazzano-mc -o yaml
kubectl get vmc tokyo -n verrazzano-mc -o yaml

Additional verifications

Navigate to the Verrazzano console, login and you should be able to see all 3 clusters:

Managed clusters in Verrazzano

Similarly, on the Rancher console, you should be able to see all 4 clusters:

Admin and managed clusters in Rancher

“local” is the Admin cluster whereas the others are the managed clusters.

Conclusion

This concludes the execise of connecting the OKE clusters deployed in different regions into a multi-cluster Verrazzano deployment. Note that in this post, we did not configure things like DNS, Certificates, Ingress Controller etc. The aim is get the multi-cluster configuration going. In a future post, we will look at those things as well.

Stay tuned for Part 3.