Increasing the number of NATs for a Kubernetes cluster

In this article we will present the latest technical challenge we have faced at Jobtome, which was solved thanks to the amazing tool of terraformer (and others), and with great caution (!).

Scenario:

At Jobtome we manage our GCP infrastructure through terraform (Have a look at our terraform GCP modules, too, if you also manage GCP through terraform!). We are in the process of transitioning everything as a code, however we obviously have some legacy component that was there before the use of terraform.

While you can import existing resources with terraform, you have to write the resource definition yourself as of now (Hashicorp says it’s something that probably in the future releases of terraform will be added).

The task:

Our task was simple: increase the number of outgoing NAT addresses for our K8s clusters. K8s is part of the ‘new’ infrastructure, so it is a seemingly simple task: in our terraform code, pass to each cluster NAT definition the new variable with the correct count.

The problem:

Applications on our K8s cluster still access legacy data stores, which wouldn’t have some IP addresses whitelisted. Since we have 5 clusters, adding in average 4 IP addresses each, it would be very painful to go on the GCP console and add that number of IP addresses manually even for only one datastore — and we have 30. Absolutely not scalable and not what we signed up for, especially if we have terraform able to do the heavy work for us.

The solution:

Notice: the original idea was to touch the resource “firewall” to allow the new IPs to communicate with the SQL instances. Turns out that there is no such thing, and that the whitelist of the IP for SQL access is an attribute within the datastore resources, so we had to import them to start managing such resources with terraform.

Here I will explain the process we followed, in short: start managing resources through terraform, then you can ‘patch up’ the NAT addresses with the data stores dynamically, and they can go up and down without any worry for scalability, monkey work, or human error.

Part I: the importing of cloudsql

Step 1: create manifests for the resources you want to adopt

This amazing non-official project called terraformer does exactly what terraform aims to do in the future. Luckily for us who use GCP, it’s made by googlers, hence GCP resource importing is a first-class citizen.

Simple command (with project names changed for security):

terraformer import google --connect=false --resources=cloudsql --projects=jobtome-one,jobtome-two,jobtome-three --regions=europe-west1,asia-east1,us-central1,europe-west6

And we got a nice folder containing three folders (one per project), each containing a file called e.g. europe-west1.tf with the content of that project in that region. Considering how terraform works, it is not a necessary organisation of the folders (and it will be refactored at one later stage), but for now it will do the trick.

Caveat 1: I do not remember if this is the actual command: for some reason I had to ask my teammates to run it and pass me the resulting folder. I was the only one out of 3 Ops members whose command was succeeding but silently doing nothing. I passed the command on slack though it disappeared because we use the free plan.

Caveat 2: the folder which I received had a lot of resources duplicated. The behaviour I described is the intended behaviour, but I received many files with many duplicated inputs. I cleaned them out by region to achieve the result intended.

Step 2: import cloudsql resources

After renaming the resources in a meaningful way, we run the command terraform import. We control the correctness by running terraform planand make sure nothing has changed.

Note: We have resources in 3 different projects, as a result we need 3 providers (for each one contains the service account to the respective project). During the terraform import, provider is given from command line!

terraform import -provider=google.legacy-europe-west1 module.cloudsql-jtm-cloud.google_sql_database_instance.airflow https://www.googleapis.com/sql/v1beta4/projects/jobtome-one/instances/airflow

Step 3: ‘patch up’ the NAT addresses to the db blocks

We are now going to add to the terraform code the “dynamic” features for which we praise terraform: in this specific case we are gonna use a ‘dynamic block’ definition inside the resource to define dynamically which are the NAT addresses.

First we add a snippet looking like that inside our datastore resource:

dynamic "authorized_networks" {
        for_each = var.nat_cluster-production-asia-east1
        content {
          name  = "access-production-asia-${authorized_networks.key}"
          value = format("%s/32", authorized_networks.value)
        }
      }

We do similarly for our 4 other clusters, making sure that we add the authorised networks only for the cluster(s) that do need access to the datastore. Thankfully dash helps me, a good snippet manager where I just type dyn_asia and the block is expanded (same for the other clusters). Know your tools, people!

After this is done, we go and clean up the NAT addresses defined statically.

We want now to run terraform plan and see nothing has changed, like usual precautionary task.

Caveat: Notice terraform will see stuff as changed because of the order of the “authorized_networks” has changed. This is nothing controllable by the user, all that we can do is to cross check that the resource removed is also re-added few lines below. The result looks something like this:

- authorized_networks {
    - name  = "access-production-europe-2" -> null
    - value = "11.22.33.44/32" -> null
  } 
...
+ authorized_networks {
    + name  = "access-production-europe-2"
    + value = "11.22.33.44/32"
  }

where the authorised network is deleted and then immediately re-added. The headache is when sometimes the deleted/added resource is not on top of the other, and you have to go up and down the terminal to find all the couples..!

Part II: expanding the NAT address pool

Step 1: create IP addresses

To have as many things as possible known beforehand (hence detecting anomalies as early as possible) we wanted to create IP addresses and add them to our DNS of an internal DNS zone. The only important thing is that the NAT routers will not adopt these IP addresses in their pool.

One particular requirement for the creation was to keep using the standard nomenclature we used so far for the existing public IP addresses; as well as that, it was important to associate the IP with ID e.g. k8s-clustername-nat5to our dns name nat5.internalzone.dev (numbering order matching 1–1).

The quick solution for that was to use scripting obviously — in this case, a terraform one. Alright, it is not truly a script, it is simply a manifest which declared resources I needed, and thanks to terraform referencing I could be sure that they were matching 1–1 between ID and DNS name. For doing that I used an empty workspace to not mix the statefile of these resources with our “real” state file. Once the resources were created, the state could be tossed anyway.

The manifest was a simple declaration of IP and DNS resource (names changed for clarity):

resource "google_compute_address" "prod-eu" {
count = local.extra_nat_for_production
name = "k8s-clustername-nat-${count.index + local.already_existing_production}"
region = "europe-west1"
}
resource "google_dns_record_set" "prod-eu-dns" {
count = local.extra_nat_for_production
name = "n${count.index + local.already_existing_production + 1}.europe-west.nat.internaldomain.dev."
type = "A"
ttl = 60
managed_zone = "internal-dev"
rrdatas = [google_compute_address.prod-eu[count.index].address]
}

And let’s not forget the most important value we want to get out of this “script”: the IP/DNS matching, and the ID of the GCP resources

output "addresses-prod-eu" {
value = {
    ip = google_compute_address.prod-eu[*].address,
    hostname = google_dns_record_set.prod-eu-dns[*].name
  }
}
output "addresses-prod-eu-id" {
value = {
    address = google_compute_address.prod-eu[*].id,
    dns = google_dns_record_set.prod-eu-dns[*].id
  }
}

The first one gives us just a heads up without going to the google console, in case it is needed later; the second one is going to be useful because we will have to import those IDs into our “main” terraform state.

Step 2: import IP/DNS resources

It is now time to add the extra NATs in our terraform manifest: we go and increase the variable address_count, then write the command terraform plan. We see that there are a lot of new resources to add, and 5 to change (the 5 NAT routers, one per cluster). The cloudstore ‘authorized_networks’ are also to change, but we will worry about this later. We see the new resources name and mark them down, because this is what we are going to use for the import.

I write a simple script now (bash) which contains a lot of terraform import: It looks more or less like this

#!/bin/bash
set -e
terraform import "google_dns_record_set.europe-west1_production_internal_dev[4]" "internal-dev/n5.europe-west1.nat.internal.dev./A"
terraform import "module.cluster-production-europe-west1.module.nat.google_compute_address.address[4]" "projects/jobtome-one/regions/europe-west1/addresses/k8s-clustername-nat-4"
…goes on for all resources

Notice how the resource array order (in the internal state) is respected thanks to the programmatical resource creation of point 1. The script is set to exit at the first error, though no errors were supposed to happen (and I commented then tried the first lines myself outside of the script, to see if it was correct. Safety is never enough). The two imports have:

• as first parameter, the name we want the resource to appear as in the state file. It was taken from the terraform plan we just launched (we needed to know only the first resource name, all the others will have incremental numbers)
• As second parameter, the ID of the GCP resource obtained by launching terraform apply of point 1. NB: you cannot launch terraform output after you toss that unimportant state file! Make sure your terminal is still open :D

Tip: for having an extra-check and be (righteously) paranoid of breaking everything, you can use the command ‘terraform state show’ and the internal name of the resource, like

terraform state show “google_dns_record_set.us-central1_production_jobtome_domain[0]”

Step 6: raise number of nat IP (fo’ real)

Now that everything is in place, we have a terraform plan and check that the only actions to perform are change operations, not adding anymore because there is no IP/DNS for terraform to add. Five resources to be changed are the NAT routers, the rest of the operations is the new authorized networks. The NAT routers have now adopted the static IP addresses that we had defined, we can see it on the console of GCP (“in use by”, which was previously empty with a yellow “!”).

We know now that this adoption won’t be a problem, because the cloudstore dynamically authorises all the cluster IP addresses, no matter if 1 or 10.

Step 7: ???

Step 8: profit!!!

Results:

We have now adopted our legacy resources into the new paradigm of IaC, and patched up perfectly with our ‘new’ architecture: when we scale/upgrade/change anything on the new resources, the legacy part is not going to break.

Closing notes:

Hope this can help you realise how amazing IaC (Infrastructure-as-Code) is, and which tools are there to support you (be it terraform, terraformer, or a simple snippet manager!).

~Fabrizio