Kubeception with CAPOCI — Cluster API for OCI — Part 1
You may have heard we open sourced Cluster API for OCI (CAPOCI). I even mentioned it in a guest post for my buddy Joe. In this post, I’m going to explain what is Cluster API and how you can potentially use it in OCI.
Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.
A little trip down memory lane
I can hear you already: why another tool to provision Kubernetes when I could have used Terraform, Pulumi or the SDKs to create one? Well, it turns out that maintaining the scripts for an evolving system as complex as Kubernetes is hard. Very hard.
Note that I’m not referring to one that provisions a managed service such as OKE. No, I’m referring to scripts that build the entire cluster. We used to maintain one until we released OKE but that was when the variations and options for Kubernetes were limited. Take for example the cluster networking sub-space. Back then, most clusters were using Flannel or Calico (73% according to The New Stack’s State of Kubernetes Ecosystem 1st Edition). Now, there are 25 of them and counting.
Further, the problem is that every combination of IaC tool and infrastructure provider/Kubernetes distribution offers a different experience. With end-users now deploying on multiple infrastructures (read cloud providers or even hybrid), this gets even more difficult to maintain. Clearly, the need to offer a consistent lifecycle experience for Kubernetes clusters, regardless of infrastructure providers, has become more than desirable.
Enter Cluster API.
What does Cluster API do?
Cluster API (CAPI) primarily manages the lifecycle of Kubernetes clusters using a declarative API. With CAPI, you can create, scale, upgrade or destroy a cluster. But more than that, you can also use Cluster API to provision your network, load balancers, virtual machines, configure your security rules and so on.
As an analogy, think of CAPI as a Java Interface. Instead of Java APIs, CAPI uses Kubernetes-style interfaces to define the needed infrastructure for a Kubernetes cluster.
Sticking with our analogy, in order to use said Java interface, we need to implement it in a class. Well, CAPI uses an infrastructure model to extend and implement support for multiple infrastructure providers. For OCI, that’s CAPOCI.
Finally, to use our Java class, we need to create an instance. That’s when you create an instance of your preferred CAPI provider in an existing cluster. You are likely to say: “Excuse me, where does this existing cluster come from now?” Please, bear with me.
WebLogic analogy
For those of you who have used WebLogic before, this will be familiar. In a WebLogic domain, there are 2 types of servers:
- an admin server running as a singleton. Usually, we do not use it to deploy our applications. Instead, it has a management application that allows an administrative user to configure and deploy various applications and services on various managed servers or clusters
- managed servers, usually grouped together in a cluster. The managed servers are usually where the applications run.
Well, CAPI is the same:
But instead of the OS, you get an infrastructure or a cloud provider; instead of Java as the runtime, you use Kubernetes and instead of the Admin and Managed Servers, we use clusters such as Management and Workload clusters respectively. Lastly, we use the admin server to deploy applications in WebLogic; the admin server does not provision managed servers whereas with the CAPI management cluster, we use it to provision other Kubernetes clusters and not deploying applications. The table below summarizes the similarities and differences between the 2 technologies.
Management Cluster
A management cluster is a Kubernetes cluster that manages the lifecycle of the Kubernetes workload clusters. In the management cluster, we create instances of the target infrastructure providers where we want to deploy our workload Kubernetes clusters.
Creating a management cluster is simple:
- Create a Kubernetes cluster somehow.
- Install your target infrastructure provider
On OCI, the easiest way to create a management cluster is to use OKE. Once it’s provisioned, obtain its kubeconfig and follow the rest of the steps below.
Install clusterctl
First, on the host where we have kubectl and kubeconfig for the OKE cluster, we download the clusterctl tool:
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.1.5/clusterctl-linux-amd64 -o clusterctlchmod +x clusterctl
sudo mv clusterctl /usr/local/bin
From this host, we must be able to reach the Kubernetes API server of OKE. You can use OCI Cloud Shell or if you use the terraform-oci-oke project, you can use the operator host. Make sure it’s in your path and verify:
clusterctl version
clusterctl version: &version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.5", GitCommit:"d488bdb875d302eba85221fa35c9f13c00df4ae7", GitTreeState:"clean", BuildDate:"2022-07-04T13:59:55Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}Next, configure the authentication. We can use either User or Instance Principal for authentication purposes. In this example, we’ll use User Principal:
export OCI_TENANCY_ID=<insert-tenancy-id-here>
export OCI_USER_ID=<insert-user-ocid-here>
export OCI_CREDENTIALS_FINGERPRINT=<insert-fingerprint-here>
export OCI_REGION=<insert-region-here>
export OCI_TENANCY_ID_B64="$(echo -n "$OCI_TENANCY_ID" | base64 | tr -d '\n')"
export OCI_CREDENTIALS_FINGERPRINT_B64="$(echo -n "$OCI_CREDENTIALS_FINGERPRINT" | base64 | tr -d '\n')"
export OCI_USER_ID_B64="$(echo -n "$OCI_USER_ID" | base64 | tr -d '\n')"
export OCI_REGION_B64="$(echo -n "$OCI_REGION" | base64 | tr -d '\n')"
export OCI_CREDENTIALS_KEY_B64=$(base64 < <insert-path-to-api-private-key-file-here> | tr -d '\n')# if Passphrase is present
export OCI_CREDENTIALS_PASSPHRASE=<insert-passphrase-here>
export OCI_CREDENTIALS_PASSPHRASE_B64="$(echo -n "$OCI_CREDENTIALS_PASSPHRASE" | base64 | tr -d '\n')"
Replace the above in bold as appropriate. Note the following:
- the private key file should be in PEM format
- the region value is the identifier of the region. You can find the list of regions and their identifiers here: https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm.
You can now initialize your management cluster as follows:
clusterctl init --infrastructure ociYou’ll see the following output:
Fetching providers
Installing cert-manager Version="v1.7.2"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.1.5" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-oci" Version="v0.3.0" TargetNamespace="cluster-api-provider-oci-system"Your management cluster has been initialized successfully!You can now create your first workload cluster by running the following:clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
This cluster is now a management cluster and you can use it to provision a workload cluster using CAPOCI.
Workload cluster
A workload cluster, according to the definition, is one whose lifecycle is managed by a management cluster. You can use CAPI to create various templates e.g. some of our customers prefer to use Ubuntu with CAPOCI instead of Oracle Linux. No problem, we have a template ready for you.
The default CAPOCI template will also define the required networking constructs such as:
- gateways
- route tables
- subnets
- network security groups
If you have an existing VCN, you can reuse it too. However, you must ensure the following:
- the network security groups have all the necessary ports opened. The required ports to open vary depending on CNIs.
- the CIDR block you specify for the VCN must not overlap with the CIDR block you specify for the Kubernetes services
- the CIDR blocks you specify for the pods running in the cluster must not overlap with CIDR blocks you specify for the Kubernetes API endpoint, worker node, and load balancer subnets
Building a compute image
Before you create a workload cluster, you need a pre-built compute image. This compute image has all the necessary Kubernetes components pre-installed and will be used to instantiate your Kubernetes nodes. Follow this guide to build your image for CAPOCI.
Once the image is created, upload into OCI Object Storage and then import it in the region where you intend to build the cluster:
Creating a workload cluster
Before creating a workload cluster, you need a ssh key. Generate one as follows:
ssh-keygen -t ed25519To create the workload cluster, run the following command:
OCI_COMPARTMENT_ID=<compartment-id> \
OCI_IMAGE_ID=<ubuntu-custom-image-id> \
OCI_SSH_KEY=$(cat /path/to/public-ssh-key) \
CONTROL_PLANE_MACHINE_COUNT=1 \
KUBERNETES_VERSION=v1.22.5 \
NAMESPACE=default \
NODE_MACHINE_COUNT=1 \
clusterctl generate cluster <cluster-name> | kubectl apply -f -Replace the values in bold above as appropriate. The compartment id is the target compartment where you want to create all the associated resources such as VCN, subnets, NSGs, compute instances, Load Balancers etc. The namespace value refers to the namespace in the management cluster where your workload cluster will be defined.
Note that you can change additional workload cluster or Kubernetes parameters.
You should see the following resources created:
cluster.cluster.x-k8s.io/simplecluster created
ocicluster.infrastructure.cluster.x-k8s.io/simplecluster created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/simplecluster-control-plane created
ocimachinetemplate.infrastructure.cluster.x-k8s.io/simplecluster-control-plane created
ocimachinetemplate.infrastructure.cluster.x-k8s.io/simplecluster-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/simplecluster-md-0 created
machinedeployment.cluster.x-k8s.io/simplecluster-md-0 createdYou can watch the provisioning:
kubectl get clusters -wNAME PHASE AGE VERSION
simplecluster Provisioning 22s
simplecluster Provisioning 31s
simplecluster Provisioning 48s
simplecluster Provisioning 48s
simplecluster Provisioned 48s
Once the workload cluster is provisioned, you can obtain its kubeconfig:
clusterctl get kubeconfig <cluster-name> -n default > <cluster-name>.kubeconfigLet’s verify we can communicate with it:
k --kubeconfig=simplecluster.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
simplecluster-control-plane-bxzkr NotReady control-plane,master 63s v1.22.5Excellent! We are able to communicate with our workload cluster.
Installing a CNI
After creating a workload cluster, a CNI must be installed for the nodes to go into ready state. For CAPOCI, we have tested both Calico and Antrea. Let’s install Calico as the CNI:
kubectl --kubeconfig=<cluster-name>.kubeconfig \
apply -f https://docs.projectcalico.org/v3.21/manifests/calico.yamlOnce the CNI is deployed, you can check the status of your nodes again:
kubectl --kubeconfig=<cluster-name>.kubeconfig get nodesNAME STATUS ROLES AGE VERSION
simplecluster-control-plane-bxzkr Ready control-plane,master 4m29s v1.22.5
simplecluster-md-0-rrjrc Ready <none> 2m52s v1.22.5
Our nodes are ready and we are almost ready to start deploying application but first, we need to install the Cloud Controller Manager.
Install Cloud Controller Manager
In order to link the workload cluster to OCI, we need to install OCI’s Cloud Controller Manager (CCM). Installing the CCM will enable your cluster to automatically create a Load Balancer when you create a Kubernetes service of type LoadBalancer.
First, create a dynamic group in OCI console and add the following rules:
So that your newly created nodes are now part of this dynamic group. The OCIDs are those of the control plane and worker nodes that were created with the cluster.
Next, create a policy and add the following statements, replacing the dynamic group name and the compartment name:
allow dynamic-group [your dynamic group name] to read instance-family in compartment [your compartment name]
allow dynamic-group [your dynamic group name] to use virtual-network-family in compartment [your compartment name]
allow dynamic-group [your dynamic group name] to manage load-balancers in compartment [your compartment name]Download the sample manifest for CCM:
curl -L https://raw.githubusercontent.com/oracle/oci-cloud-controller-manager/master/manifests/provider-config-instance-principals-example.yaml -o cloud-provider-example.yamlEdit it and replace the VCN ID and the load balancer subnet ID. You can comment the 2nd load balancer subnet since in the default template we have only 1 load balancer subnet. Similarly, you can comment the security lists since we are using NSGs.
Once the changes are done, create a secret:
k --kubeconfig=simplecluster.kubeconfig create secret generic oci-cloud-controller-manager \
-n kube-system \
--from-file=cloud-provider.yaml=cloud-provider-example.yamlChoose the CCM Release version according to your Kubernetes version:
export CCM_RELEASE_VERSION=v1.22.1Download the deployment manifests to deploy the CCM:
curl -L "https://github.com/oracle/oci-cloud-controller-manager/releases/download/${CCM_RELEASE_VERSION}/oci-cloud-controller-manager.yaml" -o oci-cloud-controller-manager.yamlcurl -L "https://github.com/oracle/oci-cloud-controller-manager/releases/download/${CCM_RELEASE_VERSION}/oci-cloud-controller-manager-rbac.yaml" -o oci-cloud-controller-manager-rbac.yaml
We can now deploy the CCM:
kubectl --kubeconfig=simplecluster.kubeconfig apply -f oci-cloud-controller-manager.yamlkubectl --kubeconfig=simplecluster.kubeconfig apply -f oci-cloud-controller-manager-rbac.yaml
Wait for it to be ready:
kubectl --kubeconfig=simplecluster.kubeconfig -n kube-system get po | grep oci
oci-cloud-controller-manager-b249l 1/1 Running 0 57sCheck the logs:
kubectl --kubeconfig=simplecluster.kubeconfig -n kube-system logs -f oci-cloud-controller-manager-b249lUTC))"
I0712 09:31:51.494114 1 leaderelection.go:258] successfully acquired lease kube-system/cloud-controller-manager
I0712 09:31:51.494580 1 event.go:291] "Event occurred" object="kube-system/cloud-controller-manager" kind="ConfigMap" apiVersion="v1" type="Normal" reason="LeaderElection" message="simplecluster-control-plane-bxzkr_94d258fe-b7ac-4e8b-b658-d83a476fdf88 became leader"
2022-07-12T09:31:51.555Z INFO oci/node_info_controller.go:74 Sending events to api server. {"component": "cloud-controller-manager"}
2022-07-12T09:31:51.555Z INFO oci/ccm.go:182 Waiting for node informer cache to sync {"component": "cloud-controller-manager"}
2022-07-12T09:31:51.556Z INFO oci/node_info_controller.go:112 Starting node info controller {"component": "cloud-controller-manager"}
I0712 09:31:51.656767 1 controllermanager.go:275] Starting "service"
I0712 09:31:51.657700 1 controllermanager.go:285] Started "service"
I0712 09:31:51.657793 1 controllermanager.go:275] Starting "route"
W0712 09:31:51.657858 1 core.go:110] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0712 09:31:51.657936 1 controllermanager.go:282] Skipping "route"
I0712 09:31:51.657946 1 controllermanager.go:275] Starting "cloud-node"
I0712 09:31:51.657716 1 controller.go:233] Starting service controller
I0712 09:31:51.658089 1 shared_informer.go:240] Waiting for caches to sync for service
I0712 09:31:51.658642 1 node_controller.go:115] Sending events to api server.
I0712 09:31:51.658769 1 controllermanager.go:285] Started "cloud-node"
I0712 09:31:51.658785 1 controllermanager.go:275] Starting "cloud-node-lifecycle"
I0712 09:31:51.659025 1 node_controller.go:154] Waiting for informer caches to sync
I0712 09:31:51.659480 1 node_lifecycle_controller.go:76] Sending events to api server
I0712 09:31:51.659560 1 controllermanager.go:285] Started "cloud-node-lifecycle"
I0712 09:31:51.663342 1 controller.go:265] Node changes detected, triggering a full node sync on all loadbalancer services
I0712 09:31:51.758226 1 shared_informer.go:247] Caches are synced for service
I0712 09:31:51.758402 1 controller.go:741] Syncing backends for all LB services.
I0712 09:31:51.758439 1 controller.go:748] Successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes
Testing CCM
Let’s test CCM by deploying an ingress controller. In order for the test to succeed, we need a load balancer created when we deploy the ingress controller.
First, add the helm repo for ingress-nginx:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm show values ingress-nginx/ingress-nginx > nginx.yaml
Edit the nginx.yaml and add the following service annotations:
service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "None"
oci.oraclecloud.com/oci-network-security-groups: "<public-lb-nsg-id>"Dry run the installation to verify you’ve set your service annotations correctly:
helm install --dry-run nginx ingress-nginx/ingress-nginx -f nginx.yamlIf you’ve done this well, you can proceed with the installation:
export KUBECONFIG=simplecluster.kubeconfig
helm install nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace -f nginx.yamlThis will create a public HTTP Load Balancer.
Check the CCM logs:
" message="Ensuring load balancer"
2022-07-12T09:34:04.117Z INFO oci/load_balancer.go:409 Ensuring load balancer {"component": "cloud-controller-manager", "loadBalancerName": "61826c0f-1f4d-4398-8d58-c2d24800a9b9", "serviceName": "nginx-ingress-nginx-controller", "loadBalancerType": "lb", "nodes": 1}
2022-07-12T09:34:04.240Z INFO oci/load_balancer_security_lists.go:96 Security list management mode: "None". Not managing security lists. {"component": "cloud-controller-manager"}
2022-07-12T09:34:04.240Z INFO oci/load_balancer.go:264 Attempting to create a new load balancer {"component": "cloud-controller-manager", "loadBalancerName": "61826c0f-1f4d-4398-8d58-c2d24800a9b9", "loadBalancerType": "lb"}
2022-07-12T09:34:05.592Z INFO oci/load_balancer.go:321 Await workrequest for create loadbalancer {"component": "cloud-controller-manager", "loadBalancerName": "61826c0f-1f4d-4398-8d58-c2d24800a9b9", "loadBalancerType": "lb", "workRequestID": "ocid1.loadbalancerworkrequest.oc1.ap-sydney-1.aaaaaaaabkuhhprwe7g43hn4ui2gff7igho3zzlbynumpn7buvyb6pecrtka"}Check the Load Balancer is created in OCI Console:
And now we can see the HTTP Load Balancer is successfully created.
Initial bootstrap cluster
What if you don’t want to use OKE as your management cluster? Well, your can use an initial bootstrap cluster instead. An initial bootstrap cluster is one which is your initial management cluster for which you can use something like kind, k3s or Rancher Desktop. The idea is to use this initial bootstrap cluster as a management cluster to provision a workload cluster which you then “promote” to a management cluster by installing CAPI into this workload cluster.
Once this step is completed, you can safely discard the initial bootstrap cluster and use your management cluster to provision your workload clusters. Ladies and gents, I give you kubeception.
Summary
I hope you enjoyed this post, which was to introduce Cluster API on OCI and how you can deploy it to create a Kubernetes cluster in OCI. Note that you can also use OKE to deploy workload clusters in other infrastructure providers. All you need to do is initialize the required infrastructure provider along with the necessary authentication parameters.
In subsequent parts, we’ll push the boundaries a bit more and use it in combination with other technologies we’ve been talking about recently on this blog to see how it can help you migrate your workloads to the cloud.
Swing by our public Slack channel to discuss!
Want to try it out? Sign up for our Free Tier if you haven’t already.
Errata: In the table of similarities and differences, I mentioned “Runs application deployed by managed servers” for Managed WebLogic servers. This should read “Runs application deployed by WebLogic servers.”
