Automatic creation of Kubernetes cluster on Binary Lane

Using Terraform and Ansible to quickly stand up a Kubernetes cluster on the Australian Binary Lane cloud provider.

This article uses the principles introduced in my OpenVPN article to create a set of private and public servers within a Virtual Private Cloud (VPC) and then using an OpenVPN server to connect to them.

Kubernetes Cluster

All the software described in this article can be found on github. Note that many of the resource names are hard coded with a variable prefix. If you want to name them differently, you will need to change them after pulling down the code.

Values to be supplied

When using the code from this article, you will need to supply a number of additional files, that are not in github:

- An ssh key pair at ~/.ssh/bl-rsa and ~/.ssh/bl-rsa.pub

- A variable file called terraform/terraform.tfvars that defines your secrets:

binarylane_email = "<your email address you use with Binary Lane>"
binarylane_api_key ="<your API key from Binary Lane>"
ssh_key = "<the digital signature of your key registered with Binary Lane>"
project_prefix = "eg"

Kubernetes Cluster

The Kubernetes cluster we will create is shown in the diagram above. It consists of three nodes in the Private Virtual Cloud (VPC), eg-k8s-master, eg-k8s-node-1 and eg-k8s-node-2. All of these servers only have connections to the private VPC subnet (10.240.0.0/16) and have no connection to the Internet.

To access these nodes, we will use an OpenVPN server (eg-open-vpn) to create a VPN tunnel to our development machine. The tunnel will use its own subnet (10.8.0.0/24) for the tunnel.

The OpenVPN server will then route connections from the development machine, over 10.8.0.0/24 to the 10.240.0.0/16 VPC subnet.

To provide external access to any HTTP/S services in your cluster, we will create an nginx reverse proxy on a gateway server (eg-gw), which has two interfaces, one to the Internet and the other to the VPC subnet.

The gateway server also provides an egress point for our private nodes. These nodes require access to the Internet as they need to download packages and upgrades.

The VPC network provided by Binary Lane includes a gateway (10.240.0.1), which provides the connectivity between all the 10.240.0.0 interfaces. It also provides a NAT interface for servers within an Internet connection.

In addition, the Binary Lane gateway can also provide other static routes. This is used to provide routing from the private nodes to our gateway server for Internet egress connections.

Infrastructure as Code (IaC)

We will create this cluster using Infrastructure as Code (IaC) from my github repository.

After pulling down the code, you will see two folders:

ansible
terraform

terraform

This folder contains the Terraform scripts that will create the VPC and servers. This is the starting point.

ansible

This folder contains the Ansible scripts that will set up your servers with the packages and applications they need.

terraform

First you need to create a set of variables that hold secrets associated with your account. Add these to a file called terraform/variables.tf :

binarylane_email = "<your email address you use with Binary Lane>"
binarylane_api_key ="<your API key from Binary Lane>"
ssh_key = "<the digital signature of your key registered with Binary Lane>"
project_prefix = "eg"

When you create Binary Lane servers, you can ask Binary Lane to automatically add ssh keys that you register with Binary Lane. In the above file, you can add the signature of the ssh key you want to use. You can find the signature under your Binary Lane account.

Once you have set up your variables, you can then create the VPC and servers with the following:

cd terraform
terraform init
terraform plan
terraform apply

You need to reply yes as required.

Once this has finished you should have one VPC and five servers. Most of my articles create Debian servers but for this exercise I use Ubuntu as Debian seems to have a problem with routing outside of the VPC.

Because Terraform only allows you to touch a resource once and cannot manage circular dependencies, there are two manual steps that you must do.

First you must create a static route for your VPC from 0.0.0.0/0 to 10.240.0.a, where a comes from your gateway server private IP address. Save the configuration. Note that this can take time to take affect.

Secondly, you need to separate the public and private interfaces on the gateway VPS. You can find this setting under the Network menu for the VPS and then under VPC Interface. Select and save:

· Dedicated Interface (Public IP traffic is on a dedicated interface)

Your infrastructure is now ready to be configured.

ansible

This folder contains the Ansible scripts that will set up your servers. Because these scripts affect connectivity, you need to execute them as separate commands.

Bootstrapping Internet Facing Servers

All the servers need to be bootstrapped. This process updates the OS, creates a non-root user and sets up SSH such that the root user cannot log in remotely for security. Once the bootstrap is complete you will only be able to log in as server-admin. Also note that the private servers cannot be bootstrapped yet because you cannot access them over the Internet and you need the OpenVPN in place first.

Now you can bootstrap the two Internet facing servers:

cd ../ansible/bootstrap
ansible-playbook bootstrap.yml --limit open_vpn,gw

OpenVPN Server

Now set up the OpenVPN server:

cd ../openvpn-server
ansible-playbook openvpn-server.yml

Whilst many automated scripts create a Certificate Authority (CA) on the OpenVPN server, I prefer to use a standalone CA. I describe this in my article on creating a CA.

Part way through the setup of the OpenVPN server you will be asked to carryout two manual actions:

1. Sign the file ~/openvpn-server.req and save the result as ~/openvpn-server.crt
2. Transfer the ca.crt file from your CA to ~/ca.crt on your development machine

After each step, press enter to continue the ansible play.

OpenVPN Client

Once your OpenVPN server is set up, you can proceed to setting up an OpenVPN client.

cd ../openvpn-client
ansible-playbook openvpn-client.yml

Like the server set up, this will ask you to sign the file ~/openvpn-client.req and save the result as ~/openvpn-client.crt.

Once it completes, it will create ~/openvpn-client.ovpn on your development machine which you should be able to use with an OpenVPN client (such as TunnelBlick) to connect to your OpenVPN server. This is explained in my article on setting up an OpenVPN tunnel manually.

After you start the connection, you should find that you can now SSH into your private servers using:

ssh root@10.240.0.x -i ~/.ssh/bl_rsa

If you have not set up a bl_rsa ssh key you will need to use the one you created instead.

It is worth ensuring that you can ssh into each Kubernetes node so that you can confirm the digital signature for each.

Gateway

The gateway provides ingress to the Kubernetes cluster as well as egress to the Internet from the private servers.

Whilst it is beyond this article to explain how to set up nginx, this Ansible script will install it and, once installed, should be accessible from your browser on your eg-gw public address.

It will also set up the server to forward packets through its firewall.

cd ../gateway
ansible-playbook gateway.yml

It is worth logging into the gateway server after this completes to make sure you can ping your private servers. Then log in to each private server and ping the gateway and a public address, say, telstra.com.

If you are following along the scripts, you will see that this play creates a netplan configuration file that defines the internal NIC, ens4. This is necessary because when you separate the NICs on a Binary Lane VPS, the image does not set up the separate interface. Whilst it is tempting to modify the netplan configuration file provided by Binary Lane (/etc/netplan/50-cloud-init.yaml) by adding in the ens4 interface, you will find that when the server is rebooted, Binary Lane re-writes this file. For this reason we create our own configuration file to be read after the Binary Lane configuration.

Bootstrapping Private Servers

Now you have access to your private servers, you can now bootstrap them:

cd ../bootstrap
ansible-playbook bootstrap.yml --limit k8s_master,k8s_node

Setting up Kubernetes

Your private servers are now ready to have a Kubernetes cluster installed on them. There are two playbooks to do this and they must be executed in a particular order:

cd ../k8s
ansible-playbook k8s.yml  --limit k8s_master
ansible-playbook k8s.yml  --limit k8s_node

You can now check that your cluster is ready for use by logging in to the eg-k8s-master node and typing:

kubectl get nodes
kubectl get pods -A

This should give you a list of your physical nodes and all your baseline pods required to run Kubernetes.

Ready for Use

Now that your cluster is ready for use, you may want to consider some of these articles on how to use it:

• Adding a postgres database
• Deploying a Spring Boot application
• Using ArgoCD

I am sure you will come up with some amazing solutions.

Tearing it all down

Once you have finished with your cluster, you can reduce costs by destroying the set up.

Remember, this will lose all your data and configurations.

Go back to the github project and enter:

cd terraform
terraform destroy

You will need to confirm the Terraform plan by typing yes.

You may also consider deleting your OpenVPN client configuration as it is no longer useful.

By following te instructions in this article, you can quickly create a new Kubernetes cluster.

Summary

In this article, we created a set of infrastructure on the Australian Binary Lane cloud provider, using Terraform.

We then set up an OpenVPN connection using Ansible to gain access to our private servers. In addition, we created an ingress and egress point to the Internet for our private servers.

Having gained access to our private servers, we then used Ansible to set up our Kubernetes cluster.

We are now ready to put that cluster to use, which will be the subject of further articles.

If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.