Automating a Verrazzano multi-cluster deployment with Terraform on OCI

Deploying and managing Kubernetes is hard. And that’s before you even get into scaling multiple clusters, deploying applications and so on. Fortunately, at Oracle we have a number of tools to assist you with both.

First, we have the Terraform OKE module. It makes deploying OKE clusters on OCI a breeze. Clone the git repo, configure your provider parameters and any additional option you want, run Terraform and you’ll have a cluster and then a bit more in a few minutes.

We also make the module reusable which means you can use it many times over to deploy multiple clusters or build your own solution on top of it. But this handles only the infrastructure piece. We want to also deploy the Verrazzano Container Platform in a multi-cluster configuration. I wrote about how to do this before but now we want to automate this as much as possible and deploy to any OCI region. So from an infrastructure perspective, this is what we want to set up:

We’ll use the module for verrazzano-multicluster. Under the hood, it reuses the Terraform OKE module, including the ability to configure RPCs in the latest release. Contrast the above to my previous post when using Submariner when I needed every cluster to talk to each other in a mesh:

I also want to place my Verrazzano Admin cluster in Sydney and a managed cluster in Sydney and Melbourne.

Configuring Terraform

Clone the terraform-oci-verrazzano module:

git clone https://github.com/oracle-terraform-modules/terraform-oci-verrazzano.git

By default, the following are set:

• the Admin’s provider is configured to be Sydney. If you want your Admin cluster to be in another region, edit the admin.tf file and change the oci provider name e.g

  providers = {
    oci      = oci.sydney # change to oci.melbourne
    oci.home = oci.home
  }

Next, we configure the admin cluster in terraform.tfvars and set connectivity:

admin_region = {
  admin_name = "admin"
  region     = "Sydney" # must match the above provider name
  vcn_cidr   = "10.0.0.0/16"
  pods       = "10.200.0.0/16"
  services   = "10.100.0.0/16"
}

control_plane_type = "private"

connectivity_mode = "star"

Now, I need to explain a couple of things about Terraform and the multi-cluster module. We have pre-configured the providers for all OCI regions as a convenience. We have also created a cluster definition in each region. However, there’s currently no way in Terraform to conditionally loop through providers (see the open issue). So by default, we don’t enable all the clusters but only the admin, Sydney and Melbourne regions as examples. We disable the other clusters by commenting them. If we don’t do that, Terraform assumes you want to use all providers and will go and query the APIs for each region. This will take a long time. Instead, we comment them.

If you want to use other regions, you need to do 2 things:

1. uncomment the clusters you want to use
2. uncomment their correspondding outputs

We’ve grouped the clusters geographically e.g. Phoenix will be found in usa.tf and Mumbai will be found in india.tf and so on. You’ll find the cluster definitions in the clusters submodules.

Next, in your terraform.tfvars or whatever method you are using to configure your input variables, you also need to ensure your cluster is enabled:

clusters = {
  # Africa
  johannesburg = false
  
  ...
  # Oceania
  melbourne = true
  sydney    = true
}

Finally, since we also need of the OCIDS e.g. those of the clusters, NSGs from the selected clusters, we disabled them in the outputs.tf in the clusters module. If you select clusters in other regions to be created, you’ll need to uncomment these manually as well. I admit, it’s a hassle and I haven’t found a good solution to this yet. But for now, it’s a comparatively small price to pay for pre-defined clusters, along with routing, RPC definitions between VCNs etc.

Creating the clusters

You will need to run Terraform twice:

• 1st run: create the infrastructure and the clusters
• 2nd run: configure the kubeconfig and generate the Verrazzano scripts

Disable the cluster configuration in the first run:

configure_clusters = false

and run Terraform apply to create the clusters:

terraform apply --auto-approve

Configure connectivity

Unlike in previous releases where you had to manually create the Remote Peering Connections, we have now added the ability to automatically configure the number of RPCs required. For the admin cluster, it requires connectivity to all:

  # admin is always connected to everybody
  remote_peering_connections = {
    for k, v in var.clusters : "rpc-to-${k}" => {} if tobool(v)
  }

But the managed clusters are trickier. As demonstrated in the Submariner post, depending on the workload, we might need each cluster to be able to talk to each other regardless of their Verrazzano’s profile. So, we want to be able to choose between a mesh and a star configuration and create the RPCs accordingly:

remote_peering_connections = var.connectivity_mode == "mesh" ? { 
  for k, v in merge({"admin"=true},var.clusters) : "rpc-to-${k}" => {} 
  if tobool(v) && k != "melbourne" } : { "rpc-to-admin" : {} 
}

You don’t need to do those 2 changes, we’ve already done that for you. All you need to do is choose your connectivity mode e.g.:

connectivity_mode = "star"

Once you peer the RPCs, your network visualizer in the OCI console should look something like this:

Install Verrazzano

When installing Verrazzano, you can configure it to use DNS. This is done for you via external-dns which Verrazzano pre-installs and configures for you. If you don’t configure your own DNS, Verrazzano will automatically use nip.io. In the terraform module, you can set the configuration:

configure_dns      = false

If you want to configure your own DNS, you need to change the above to ‘true’. In addition, using the oci.yaml.example file, create the content and save it in a secret in OCI Vault:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1..aaaa
  user: ocid1.user.oc1..aaa
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    aaaa
    ....
    bbbb
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:ab:34:cd:56:ef:78:ab:90:cd:

The above user must have the necessary rights to create DNS records in OCI. Once you have created the secret, copy the secret ocid and configure DNS in Terraform:

dns_compartment_id = "ocid1.compartment.oc1..."

dns_zone_id   = "ocid1.dns-zone.oc1..."
dns_zone_name = "my.domainsecret_id = "ocid1.vaultsecret.oc1..."

This will be used later by Verrazzano. Once the clusters have been created, change the install_verrazzano parameter to true:

install_verrazzano = true

This last step will upload all the generated scripts to the operator host in the admin region.

Install Verrazzano

Set the ‘install_verrazzano’ to true:

install_verrazzano = true

And run Terraform again to upload a number of scripts to automate the installation of Verrazzano.

Obviously, if you change your regions or have more clusters, you need to amend your cluster list in the scripts below.

First, install the Verrazzano Operator in each cluster:

cd /home/opc/vz/operator

for cluster in admin sydney melbourne ; do
  bash install_vz_operator_$cluster.sh
done

Check if the operator has successfully installed:

for cluster in admin sydney melbourne ; do
  bash check_vz_operator_$cluster.sh
done

Obtain the the yaml for DNS:

cd /home/opc/vz/oci
bash  get_oci_secret.sh
echo >> oci.yaml

And create the secret in each cluster:

for cluster in admin sydney melbourne ; do
  bash create_oci_secret_$cluster.sh
done

Note that you can also use the instance_principal method but there’s an issue which we haven’t yet implemented in the Terraform OKE module which would make using this feature easier. Regardless, I was mightily pleased that the lovely folks at external-dns finally approved and merged our PR to have instance_principal support.

Install Verrazzano in Admin and managed clusters

Install Verrazzano in admin:

cd /home/opc/vz/clusters
bash install_vz_admin.sh

The admin Verrazzano cluster usually installs a bigger set of components. So, while this is going on, you can also install Verrazzano in the managed clusters in parallel:

for cluster in sydney melbourne ; do
  bash install_vz_cluster_$cluster.sh
done

Wait for Verrazzano to be successfully installed in all clusters:

bash vz_status.sh

# Admin cluster takes longer, check separately
ktx admin
kubectl wait --timeout=20m --for=condition=InstallComplete verrazzano/admin

Registering the managed clusters

Once Verrazzano has been successfully installed in all clusters, we need to register the the managed clusters. First, create the certificates secrets:

cd /home/opc/vz/certs
for cluster in sydney melbourne; do
  bash create_cert_secret_$cluster.sh
done

Create a ConfigMap for the API Server:

cd /home/opc/vz/cm
bash create_api_cm.sh

And then create the VerrazzanoManagedCluster CR for each cluster:

cd /home/opc/vz/clusters
for cluster in sydney melbourne ; do
  bash create_vmc_$cluster.sh
done

We can now register all clusters:

for cluster in sydney melbourne ; do
  bash register_vmc_$cluster.sh
done

Accessing Verrazzano

Run vz status to get the urls:

kubectx admin
vz status

Login to the Rancher console and you’ll see the Sydney and Melbourne clusters registered:

Let’s login to Argo CD, navigate to the Settings and then the Clusters page:

We can see both clusters are registered. We can now deploy multi-cluster applications using either the OAM approach or using the GitOps approach with Argo CD which we’ll explore in a future post.

Let’s check the new Verrazzano monitoring dashboards in Grafana:

We can similarly see the metrics for each cluster getting captured.

Summary

In the latest release of the Terraform OKE module, we’ve added the ability to dynamically configure the number Remote Peering Connections. They can be created for admin-managed (star architecture) or a mesh architecture.

We then reused the Terraform OKE module to create a higher level module that creates an Admin Verrazzano cluster as well as 2 managed Verrazzano clusters. Finally, we generated installation and registration scripts for Verrazzano multi-cluster to automate the whole process.

As Verrazzano gains more adoption, we encounter more interesting use cases and it’s important we can configure things at scale in an automated way. This new multi-cluster module for Verrazzano on OCI is the first step in giving our users the ability to implement those use cases.

I hope you enjoyed this article. If you have ideas to improve the module or interesting use cases, do not hesitate to bring them to our attention. We would love to hear from you.