Announcing release of Terraform OKE (Kubernetes) module 5.0 — Part 3
Since the very first commit of the Terraform OKE module, we have had module reusability in mind. We wanted users to be able to use it to create their clusters or to extend it for their own use cases by adding their own customizations.
At times, we were limited by HCL. We either had to pass a large number of variables or reduce these in an attempt to make it simpler for the user by passing related variables as complex types. The latter approach introduced an element of inflexibility and we went back to primitive types or collections of primitive types for which Terraform had greatly improved support in the form of additional functions. At other times, Terraform’s module implementation was itself limited e.g. modules were not conditional before but they can be now. As Terraform has improved, so has our ability to make the Terraform module more flexible and thus more reusable without reinventing the wheel.
In this article, after explaining how we’ve made the OKE module more flexible for your workload, it’s time we examine its reusability.
Ways of reusing
There are 4 ways of (re)using the Terraform OKE module:
- By cloning the repo directly, extending and/or modifying the code
- By using as a remote module from the HashiCorp registry where it is also published
- By using as a remote module directly from GitHub
- By cloning as a submodule in your own git repo
Let’s examine these in turn.
Using the module by cloning the GitHub repo
This method is probably the simplest and it’s where most users start because it allows you take the module and OKE for a spin. You clone the GitHub repo:
git clone https://github.com/oracle-terraform-modules/terraform-oci-oke.git tfokeConfigure your providers:
provider "oci" {
fingerprint = var.api_fingerprint
private_key_path = var.api_private_key_path
region = var.region
tenancy_ocid = local.tenancy_id
user_ocid = local.user_id
}
provider "oci" {
fingerprint = var.api_fingerprint
private_key_path = var.api_private_key_path
region = var.home_region
tenancy_ocid = local.tenancy_id
user_ocid = local.user_id
alias = "home"
}and your input variable file(s) and you run:
terraform applyThe advantage of this method is that you get the latest and greatest. This method also works great if you would like to contribute back to the project. We appreciate any kind of contribution whether this is in the form of code contribution, features, bug fixes, testing, documentation or just simply bringing your use case by creating issues. All you need to do is sign the Oracle Contributor Agreement (creating issues on GitHub does not need OCA) and you can start contributing.
Some users also tended to use this method to clone and change the module for their own needs, especially with the previous flexibility and reusability limitations. With this method, they have the greatest power to change it. As always, with great power comes great responsibility (Ben Parker dixit) and this was done with the conscious acceptance that once they changed, they won’t be able to get the latest updates and maintenance is entirely their own. Or you could still do it but not without some git wizardry. Hence, this is not a path to be taken lightly.
Using as a remote module from Terraform registry
Instead, using the OKE module as a remote Terraform module from the Terraform registry is the method I would recommend, especially if you are getting started and you need to run a real environment with actual workloads. It’s the easiest and you can keep track with new features, bug fixes with new releases. Below is an example of using it as a remote module:
module "oke" {
source = "oracle-terraform-modules/oke/oci"
version = "5.1.0"
home_region = local.admin_region
region = local.admin_region
tenancy_id = var.tenancy_id
# general oci parameters
compartment_id = var.compartment_id
....
}The OKE module has a quite lengthy list of input variables. This is not to make things complicated but rather reflects the extraordinarily large and sometimes diametrically opposite use cases of our users and we want to meet them all. To make things easy, we pick some defaults but we allow you to switch things around to suit your needs e.g. the control plane can be public or private, as can the workers and load balancers and you can have any combination of them. This reminds me of Ian Malcolm played by Jeff Goldblum in Jurassic Park:
Hence the reason we also pick some sane defaults for you. So you don’t need to trawl through the lengthy list, you’ll be pleased to know the input variables are also now organized by submodules.
With this method, you can build your own cloud native projects on OKE using known releases as a baseline and also upgrade at your own pace. You can add your own extensions and favorite Kubernetes utilities, deploy your helm charts to OKE and so on. We already include some of these as community extensions and you can take a look at them to see how you can add your own.
Using as a remote module from GitHub
The Terraform registry will only update when we cut a new release. If you like the remote module method but would like to stay close to the latest and help us test so you can be ready to adopt a new release, you can also point to GitHub directly instead. For example, let’s say we are working on a new feature in a branch and you would like to test how this new feature impacts your infrastructure, you can do so like this:
module "oke" {
source = "github.com/oracle-terraform-modules/terraform-oci-oke?ref=featurebranch&depth=1"
home_region = local.admin_region
region = local.admin_region
tenancy_id = var.tenancy_id
# general oci parameters
compartment_id = var.compartment_id
}Replace the featurebranch above with the branch name where we are testing the new feature but have not merged with main yet. As an example, a number of our users helped us test the v5 release by using the 5.x branch before it was ready for merge with main. We’ll likely keep this type of branch around for a while especially for important and complex changes to allow for a more thorough testing until we are satisfied. However, you should not plan to run your active clusters using this method. Once the feature branch is merged, we’ll very likely delete it for housekeeping purposes.
Clone as a submodule in your own git repo
The OKE module is implemented in a sort of micro-services architecture and leverages other Terraform OCI modules such as VCN, Bastion, Operator and DRG that are published on the Terraform registry. These other modules perform 1 function only and like the OKE module, they are also reusable and when used together, composable. When you run terraform init, you’ll see the Terraform client download them from the Terraform registry.
tf init
Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/oracle-terraform-modules/drg/oci 1.0.5 for drg...
- drg in .terraform/modules/drg
- extensions in modules/extensions
Downloading registry.terraform.io/oracle-terraform-modules/vcn/oci 3.6.0 for vcn...
- vcn in .terraform/modules/vcn
- vcn.subnet in .terraform/modules/vcn/modules/subnet
...However, some of our users may have additional constraints e.g. their target OCI region is one of the Government Cloud regions (lucky you) and as a result may have to operate in an air-gapped environment.
Operating in such an environment means they cannot download any dependent sub-module from the Terraform registry. Hence, instead of operating like a micro-services application, the OKE module must be entirely self-contained. You can read more about self-contained systems here or watch this great talk (albeit not necessarily about Kubernetes or cloud infrastructure but I think it will convey the idea). By embedding the OKE module into your own using git submodules, you can then avoid the need for the Terraform client to call the registry and fetch the OKE and its sub-modules.
Reusing selective parts: Networking (VCN, subnets, DRG)
I mentioned above that the OKE module consists of a number of sub-modules. In the v5 release, we’ve made it possible to almost entirely pick and choose which parts you need created or enabled. By default, the entire infrastructure will be created for you:
- a VCN
- subnets for control plane, workers and load balancers
- subnets for bastion and operator hosts if they are enabled
- NSGs and default security lists
- worker nodes and so on.
Let’s say you already have a VCN and want to reuse it and let the module create the subnets for you. All you need to do is change the following variables:
create_vcn = false
vcn_id = "ocid1.vnc.oc1..aaa....."
subnets = {
bastion = { newbits = 13, netnum = 0, dns_label = "bastion" }
operator = { newbits = 13, netnum = 1, dns_label = "operator" }
cp = { newbits = 13, netnum = 2, dns_label = "cp" }
int_lb = { newbits = 11, netnum = 16, dns_label = "ilb" }
pub_lb = { newbits = 11, netnum = 17, dns_label = "plb" }
workers = { newbits = 2, netnum = 1, dns_label = "workers" }
pods = { newbits = 2, netnum = 2, dns_label = "pods" }
}It’s possible your existing VCN may be used by your team mates, with other workloads already running in existing subnets. As such, it’s a good idea to use the newbits and netnum cidrsubnet parameters to set the boundaries of your OKE subnets so they don’t overlap with existing subnets. If you’re not familiar with these parameters, you can also use the CIDR block format instead to create the subnets required by OKE:
subnets = {
bastion = { cidr = "10.0.0.0/29" }
operator = { cidr = "10.0.0.64/29" }
cp = { cidr = "10.0.0.8/29" }
int_lb = { cidr = "10.0.0.32/27" }
pub_lb = { cidr = "10.0.128.0/27" }
workers = { cidr = "10.0.144.0/20" }
pods = { cidr = "10.0.64.0/18" }
}I tend to use the cidrsubnet method because it’s also easier to calculate subnet size and create contiguous subnets. Also, when reusing an existing VCN created by somebody else, you may not have the luxury of wasting IP addresses so using this method together with ipcalc, is an excellent way to work out your subnets boundaries.
Let’s now say these subnets were already created and you want OKE to use the pre-created ones. Instead of CIDR blocks, you can provide their OCIDs:
subnets = {
operator = { id = "ocid1.subnet..." }
cp = { id = "ocid1.subnet..." }
int_lb = { id = "ocid1.subnet..." }
pub_lb = { id = "ocid1.subnet..." }
workers = { id = "ocid1.subnet..." }
pods = { id = "ocid1.subnet..." }
}OKE will then use them to deploy its various components.
Some users want to deploy their OKE clusters as part of a wider infrastructure e.g. using a hub-and-spoke architecture. If both your hub and spoke VCNs are in the same region and you want them to share the same DRG, all you need to do is set the following:
create_vcn = true
create_drg = false
drg_id = "ocid1.drg....."A new VCN will be created for you and it will be attached to the existing DRG. You can then further configure routing rules:
nat_gateway_route_rules = [
{
destination = "192.168.0.0/16" # Route Rule Destination CIDR
destination_type = "CIDR_BLOCK" # only CIDR_BLOCK is supported at the moment
network_entity_id = "ocid1.drg....." # for nat_gateway_route_rules input variable, you can use special strings "drg", "nat_gateway" or pass a valid OCID using string or any Named Values
description = "to hub"
},
]If they are in separate regions, each VCN must then have its own DRG:
create_drg = true
nat_gateway_route_rules = [
{
destination = "192.168.0.0/16" # Route Rule Destination CIDR
destination_type = "CIDR_BLOCK" # only CIDR_BLOCK is supported at the moment
network_entity_id = "drg" # for nat_gateway_route_rules input variable, you can use special strings "drg", "nat_gateway" or pass a valid OCID using string or any Named Values
description = "to spoke"
},
]The network_entity_id can take the values “drg” which will configure the additional routing rules to use the DRG created by the OKE module itself. All you have to do is establish the Remote Peering Connection (RPC) between the VCNs of the 2 regions.
Note that you can also use other network resources e.g. private IP address OCID. This is useful if you want your traffic to go through a specific firewall before.
Reusing selective parts: bastion host and operator
The OKE module also includes 2 extra hosts:
- bastion
- operator
We use the bastion host only for ssh jumping. This allows us to keep the bastion as lightweight and cost-effective as possible and since it’s going to be operating at the edge, we also want to limit its attack surface area and not run anything else. If you intend to use the bastion for more than just ssh jumping e.g. for data transfer, you can also change its shape:
bastion_shape = {
shape = "VM.Standard.E4.Flex",
ocpus = 4,
memory = 4,
boot_volume_size = 50
}Changing the shape allows you to also change its memory and network bandwidth and process higher amounts of data at faster speed.
Other post-provisioning operations such as installing extensions, utilities are done on the operator. There are many reasons for doing this and they are too lengthy to elaborate here. Enabling them only requires you to set their respective variables to true:
create_bastion = true
create_operator = trueSome users have alternate connectivity mechanisms in place (VPN, FastConnect, WireGuard) and these negate the need for a bastion host. In this case, just set the bastion to false:
create_bastion = falseand you should then be able to access the cluster or the operator directly, even if the control plane is private. Others, despite existing connectivity via FastConnect or VPN, they still like the idea of having a bastion host and view it as an extra layer of security but do not want to make it accessible on the public internet. In this case, you can also configure the bastion host to be private:
bastion_is_public = falseReusing selective parts: cluster
Users also want to have a say about cluster creation. Most users want to create a new cluster but there are others, depending on their governance model, who may only be able to reuse an existing cluster. To create a new cluster (this is the default behavior):
create_cluster = trueTo reuse an existing cluster, you must then also provide its OCID:
create_cluster = false
cluster_id = "ocid1......"Typically, you use an existing cluster to add more worker nodes.
Reusing selective parts: worker pools
Similarly, users would like to control the creation, types and many other characteristics of their worker nodes. See the earlier Part 2 of this series for a more in-depth examination of worker pools and how we have improved your capability to create worker nodes.
Reusing selective parts: extensions
In v5, we have expanded the extensions and separated extensions from utilities. Extensions are things you add to the cluster after its creation. Examples of extensions include the cluster autoscaler, advanced networking using other CNIs such as Multus, RDMA (using OCI Cluster Networks), SRIOV and Cilium as well as metric server among others. Most of these extensions are provided as convenience. If there’s something you use frequently and you would like added as a convenient extension, please send us a PR or open an issue for us to consider. Sometimes we may not be able to accept it. In this case, you can always add it to your own fork.
Reusing selective parts: utilities
The early versions of the OKE module had only 3 utilities (along with helpful aliases) which we installed on the operator host:
- oci
- kubectl (k)
- helm (h)
Since then, a large number of Kubernetes utilities have sprung up like mushrooms after rain. More recent community additions to the OKE module include kubectx and kubens as well as k9s. To enable them, set the following:
operator_install_k9s = true
operator_install_kubectx = truekubens allows you to change the default namespace easily so you don’t need to specify the target namespace with every kubectl commands. So instead of:
kubectl -n kube-system get pods
kubectl -n monitoring logs -f <podname>you just change the default namespace (kns is the alias for kubens):
# list the namespaces
$ kns
default
kube-node-lease
kube-public
kube-system
network
# changes active namespace
$ kns kube-system
Context "oke" modified.
Active namespace is "kube-system".
# run kubectl in kube-system namespace without specifying it
$ k get pods
NAME READY STATUS RESTARTS AGE
coredns-7fc4ffbc6f-lps6g 1/1 Running 0 7m4s
coredns-7fc4ffbc6f-mgnhk 1/1 Running 0 7m4s
coredns-7fc4ffbc6f-ssb7z 1/1 Running 0 11m
csi-oci-node-5t584 1/1 Running 1 (7m11s ago) 8m3s
csi-oci-node-dhjpz 1/1 Running 1 (7m15s ago) 8m5s
csi-oci-node-wvlrm 1/1 Running 0 8m6s
kube-dns-autoscaler-594796dfcd-6gwnz 1/1 Running 0 11m
kube-flannel-ds-h27fn 1/1 Running 1 (7m14s ago) 8m5s
kube-flannel-ds-lrzgs 1/1 Running 0 4m3s
kube-flannel-ds-wnjfr 1/1 Running 1 (7m11s ago) 8m3s
kube-proxy-hvgz4 1/1 Running 0 8m3s
kube-proxy-kxjth 1/1 Running 0 8m5s
kube-proxy-nvxpw 1/1 Running 0 8m7s
proxymux-client-6lpnm 1/1 Running 0 8m3s
proxymux-client-b6d5d 1/1 Running 0 8m6s
proxymux-client-xpx4q 1/1 Running 0 8m5s
# tail a pod in kube-system namespace without specifying it
$ k logs -f kube-flannel-ds-lrzgs
....
I1203 02:12:27.953099 1 kube.go:126] Waiting 10m0s for node controller to sync
I1203 02:12:27.957847 1 kube.go:452] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.1.0/25]
I1203 02:12:27.957891 1 kube.go:452] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.0.0/25]
I1203 02:12:27.957903 1 kube.go:452] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.0.128/25]Similarly, if you don’t want to type the commands, you can just use the k9s client to navigate:
or to stream the logs from a pod:
Another useful utility is kubectx (alias ktx) and we’ll look at how to use it in the next section.
1 cluster to rule them all: a multi-region, multi-cluster example
OKE users deploy Kubernetes clusters in interesting ways:
- single, isolated clusters
- multiple clusters connected to a hub which frequently also runs a management cluster
- multiple clusters connected to each other in a mesh
For the single and isolated type of deployment, you can use the Terraform OKE module and rinse and repeat as many times as you need. However, for those who deploy clusters connected to a hub or in a mesh, having a bastion and operator pair for each seems like a waste of resources. In this case, what we need is the ability to control and operate multiple clusters from 1 operator. As these clusters can be in the same or different VCNs and regions, let’s lay down our infrastructure needs:
- connectivity and routing between VCNs through DRG
- Remote Peering Connections if the clusters (and VCNs) are in different regions
- connectivity mode: either to the hub cluster or in a mesh. This impacts your routing rules.
- only 1 bastion and operator pair to operate all the clusters
You can take a look at the Terraform module for Verrazzano where we configure all these. As we are handling multiple clusters, we must be able to change Kubernetes context and possibly configure different software packages and we can do so easily with kubectx:
# set context to the admin cluster and install the Verrazzano Admin profile
ktx admin
...
# set context to the sydney cluster and install the Verrazzano managed-cluster profile
ktx sydney
...
# set context to the melbourne cluster and install the Verrazzano managed-cluster profile
ktx melbourne
...If you set the control planes to public, the communication although encrypted will be over public internet. To improve your multi-cluster security posture, you can peer the VCNs using Remote Peering Connections (RPCs). In this way, the communication will happen over OCI’s backbone.
As the VCNs will then be connected, their CIDRs must not overlap with each other and yet the clusters must be able to route traffic to other VCNs. In this case, you must specify additional routing rules:
nat_gateway_route_rules = [
{
destination = "10.1.0.0/16" # Route Rule Destination CIDR
destination_type = "CIDR_BLOCK" # only CIDR_BLOCK is supported at the moment
network_entity_id = "drg" # for nat_gateway_route_rules input variable, you can use special strings "drg", "nat_gateway" or pass a valid OCID using string or any Named Values
description = "To cluster 2"
},
]and you can repeat vice-versa in cluster2’s definition:
nat_gateway_route_rules = [
{
destination = "10.0.0.0/16" # Route Rule Destination CIDR
destination_type = "CIDR_BLOCK" # only CIDR_BLOCK is supported at the moment
network_entity_id = "drg" # for nat_gateway_route_rules input variable, you can use special strings "drg", "nat_gateway" or pass a valid OCID using string or any Named Values
description = "To cluster 1"
},
]Peering the VCNs of 2 clusters is easy as you can see from the above. All you need to do is configure the above input variables and set the CIDR block of your other VCN.
What if you are setting up a multi-cluster infrastructure in multiple regions and you need the pods in each cluster to be able to communicate to each other? I wrote about 1 such scenario and what you therefore need is the ability to create a variable number of RPC connections.
remote_peering_connections = var.connectivity_mode == "mesh" ? { for k, v in merge({ "admin" = true }, var.clusters) : "rpc-to-${k}" => {} if tobool(v) && k != "melbourne" } : { "rpc-to-admin" : {} }Once the secure connection and routing is configured, you can then use higher level software such as Istio and/or Submariner so that the pods in each cluster can communicate with each other.
Summary
In this article, we examine the reusability features of the OKE module in v5 and how you can use them to improve your productivity as well as deploy advanced and resilient cloud native infrastructures in a secure manner. Whether you want to deploy single isolated clusters or multiple clusters to deploy Active-Active, Active-DR or in a mesh, the Terraform OKE module can help you reach there faster.
In the final part, we’ll examine the improved robustness of the Terraform OKE module.
