“gray spacecraft taking off during daytime” by SpaceX on Unsplash

Kubernetes on Vsphere (Terraform — Part 2)

Kubernetes on Vsphere (Terraform — Part 2)

In the part 1 of this serie, I talked about Packer and how I could create templates of virtual machines from a single and unified source of truth (a single file called … a template). I described how Packer was deeply integrated into the Vsphere platform and how I could automate the complete pipeline: going from one single file describing the VM to a working VM template already uploaded on the platform.

At the end of the previous part, you should see something like this in your Vsphere:

Your newly created template in the Vsphere

I use the build timestamp to name the templates

Now that we have this perfect template, we should be able to use it to deploy as many virtual machines as possible with the exact same configuration. Although this is true in theory, we still have a lot to do in practice, believe me.

Besides, let’s not forget that the ultimate goal is to create a kubernetes cluster on Vsphere. This will also ask for additional work.

Terraform: infrastructure as code

Just like I explained in the previous article, we do not want to click on buttons to get work done. This does not scale well, it takes a lot of time and it is prone to errors. We want a single source of truth for each VM we are going to deploy; this means a file that is describing VM’s deployment and a way to transform this file into an infrastructure artifact. Terraform can perform those tasks and is the way to go.

Terraform and Packer are both edited by Hashicorp and Terraform comes right after Packer in the pipeline

Unfortunately, this also means another templating engine that you need to learn. Hopefully, it is not that complicated and it offers a very high level of flexibility and customization.

Building the template

You should start by creating a directory that is going to be your working space. Then, create 3 files in this directory:

• variables.tf: variables definitions
• terraform.tfvars: actual values of variables
• node.tf: deployment template

Terraform variables

The first one is named variables.tf and holds the different variable definitions that are going to be used in this deployment. This file does not hold the actual values but is important so that we maintain a strict definition of those variables. Remember that another strength of having those files as source of truth is to be able to track them with git and to offer a seamless collaboration in the team:

Variable definitions

Yeah, lots of definitions but these are mandatory. For example, choosing the wrong vsphere_vm_folder in the kubernetes deployment will make the process fail miserably.

Actual variable values

The second file is named terraform.tfvars and holds the variable values. You should see this file as a basic env.bash file. The main difference is that you are only able to use variables actually defined in your variables.tf file:

Replace with your values

Note that you do not need to use the export keyword like you would do in bash

Terraform template

The last file is named node.tf and is the template describing the deployment (using the previously defined variables). This template is split into several sections. We are going to deep dive in some of them.

Terraform provider
This is where you choose the underlying cloud and the authentication configuration (here: credentials). In our case, this is of course Vsphere. From their documentation:

Providers are responsible in Terraform for managing the lifecycle of a resource: create, read, update, delete.

Most providers require some sort of configuration to provide authentication information, endpoint URLs, etc.

Terraform provider for Vsphere

Note that we use directly the previously defined variables. Once you have chosen your provider, you have access to a wide range of options to describe your deployments (network, storage, cpu, etc…).

Terraform template
Remember the template we created with Packer? This is the moment to use it! The path to the template depends on where you uploaded the file with Packer. Normally, it should be in the VM folder:

Terraform template

Virtual machines
Now, it is actually time to configure the VM itself. We use Gcloud’s configuration of virtual machines. Our VM can be seen as a n1-standard-2. Of course, you can choose your own configuration. To make a choice, you should check this documentation: you must carefully choose the size of your master node because it will impact the amount of nodes you will be able to run in your cluster. In our case, we can have between 6 and 10 nodes. If we want to scale the cluster, we have to scale the master as well. This is the configuration of the VM:

VM configuration

There are several things that we need to explain here:

• We have defined a count variable inside the configuration file. This variable will be used by other definitions (we will be able to create multiple VMs with one deployment file).
• name is using the count variable to name the VM at runtime. For example, if we decide to create 3 VMs, they will be named node-1, node-2 and node-3. This is why the Terraform templating engine is so convenient.
• We have to set enable_disk_uuid to true because this is important for our kubernetes cluster to run properly (more on that later).
• wait_for_guest_net_timeout is set to 0 so that Terraform does not wait for the VM to get an IP address. We do not need this because we use DHCP and it takes some time to get an IP.
• We directly set the hostname at runtime using the count variable. This is important that nodes get a different hostname otherwise you won’t be able to deploy kubernetes.

For variables defined in the terraform.tfvars file, there are some choices that are really important. You need, for example, to group your nodes into the same VM folder and into the same pool of resources. Indeed, as we will see later, during kubernetes deployment, we will run some workloads in the cluster to perform Vsphere configuration. During this step, we will carry out some checks and if nodes are not setup correctly, it will fail. In our case, I decided to group them into a kubernetes folder:

vsphere_pool = "kubernetes"
vsphere_vm_folder = "kubernetes"

Here is the complete node.tf file for this deployment:

Full configuration

Making the deployment

You first need to init your Terraform directory by running the following command:

terraform init

This should download the Vsphere plugin.

Note that you could perfectly decide to provision another platform (they have plugins for most of the cloud providers)

The next command allows you to verify what are the different operations needed to meet the state described into your deployment files:

terraform plan

In our terraform.tfvars file, we set the variable count to 2. It means that after the plan command, you should see something like this:

Terraform is planning the deployment

As you can see, Terraform understood that we wanted to created two artifacts in our infrastructure. To perform the deploy, you should run:

terraform apply

Terraform will create your artifacts from the Packer template and customize according to what you described in your file. In the end, you should see something like this:

Success!

You should be happy right now because you have the building blocks of your future kubernetes cluster! You can actually check your Vsphere and see the two nodes that we just created:

Nodes are named correctly

Because we carrefully configured our template with Packer, you should see that VMs have already asked for an IP to the DHCP and that the VMware tools are up and running on all the different machines:

IP from DHCP, a working VMware tools and a DNS name

Normally, Terraform has created a new file (terraform.tfstate) containing the actual state of your deployment. By sharing this file, you offer the possibility to manage deployment’s resources. The only thing to do is to track with git those files, and share in a vault the credentials (terraform.tfvars):

Deployment’s state is a simple JSON file

If you want to destroy the VMs, you can just run:

terraform destroy

Going further

Terraform is awesome. But actually, we just experienced a small part of its awesomeness. We will present in this section additional actions that you can perform with this great tool.

Scale your deployment

We defined our count variable and decided to create two different nodes. You can actually change this value in terraform.tfvars and ask Terraform to apply this modification. In the next example, we increase to 3 the count value:

terraform plan

Terraform plans to create another machine

If you run the apply command, you see that Terraform creates another machine and names it node-3:

Scaling up our deployment

You could also later decide to scale down your deployment. This can be done in a similar fashion.

Interacting with the machines

You can interact with the machines after they are created. You can define output in the deployment file so that you get some information. From their documentation:

Outputs define values that will be highlighted to the user when Terraform applies, and can be queried easily using the output command

In this example, we use output for a very specific use-case: getting back IP addresses of our newly created machines. This is particularly convenient when you want to fully automate your deployments or when you want to perform post-processing tasks directly on the machine. Append this little piece of code at the end of our node.tf file:

We define an output

Then, in order for Terraform to synchronize with the new configuration, run the following command:

terraform refresh

You should get in your console the IP addresses of your machines.

Our different outputs

Awesome isn’t it? Later, you can get those addresses by running:

terraform output

Or, you can directly parse the JSON file that has been updated after the refresh command.

Getting back those IP addresses is quite important for our kubernetes deployment. It will allow us to automatically connect to our machines and start a master or a node and connect it to the existing cluster. We have actually created a command line interface to easily scale our kubernetes cluster.

Using modules

A module is like a docker image. Indeed, from their documentation:

Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. Modules are used to create reusable components in Terraform as well as for basic code organization

I have not used modules yet but it seems to be a really great idea. This module abstraction is also coming with the Terraform Registry which has the exact same role as the docker hub for the docker platform. This could facilitate the use of different clouds to a great extent. Indeed, some providers are already creating modules to allow the community to easily manage their resources from their cloud. Check out this example for the google cloud platform (for a SQL database).

Using Packer and Terraform, we were able to create the building blocks of our future kubernetes cluster. As you can see, those tools allowed us to achieve this difficult task with code and only code.

In the next and final article, we will see how we can use those machines to come up with a working kubernetes cluster!