High availability NAT gateway at Google Cloud Platform with Cloud NAT

Last week Google made a great announcement: Cloud NAT. Although the service is still in beta (see the GCP’s launch states), we couldn’t help to give it a try.

But what is Cloud NAT and why are we so excited about it? Cloud NAT is a regional service managed by Google that allows our instances to connect to the Internet even if we don’t assign them a public IP (ephemeral or dedicated) through NAT outbound (Cloud NAT does not implement inbound NAT).

Cloud NAT (source: https://cloud.google.com)

Until now, the only alternative to this was setting up our own high availability NAT in Google Cloud Platform (see our article about how to set it up that is still valid until Cloud NAT will be in General Availability stage).

The main benefits of Cloud NAT are:

• Managed Service: we can set it up in minutes and all the components are managed by Google.
• Security: we can protect our platform by only allowing access to external resources without exposing to the Internet.
• High Availability: Google provides high availability service without human intervention.
• Scalability: the service automatically adds the required resources to accommodate the traffic originated for our instances.
• Logging: still in development but it will enable us access to the traces related to the service.

Internally, Cloud NAT is a software-defined service, not an instance or appliance-based solution. Although Cloud NAT uses the Cloud Router resource to group the NAT gateway configurations, Cloud NAT is not actually performed by the Cloud Router. The NAT is made by Andromeda, Google’s network virtualization stack, that allows that an instance that uses Cloud NAT has as much external bandwidth as a VM with an external IP and the NAT gateway does not affect Cloud Router performance.

Having said that, let’s create our first Cloud NAT.

The diagram of the solution is:

Solution schema

First of all, we are going to create a new custom VPC called “custom-network1” with two subnets for the us-central1 region:

• subnet-us-central-dmz: we want to allow Internet access through Cloud NAT (192.168.1.0/24)
• subnet-us-central-backend: we don’t allow Internet access (192.168.2.0/24)

VPC setup

Then we are going to create three Debian instances:

• bastion-1: instance with an ephemeral public IP (in production we’ll need a reserved public IP but for our test is not required) inside subnet-us-central-dmz.
• nat-test-1: instance with no public IP inside subnet-us-central-dmz for which we are going to allow Internet access.
• backend-1: instance with no public IP inside subnet-us-central-dmz.

VM instances

The next step is the creation of a firewall rule to enable us ssh access to the instances

Firewall rule to allow ssh access

Now, we are going to connect to our bastion host and jump to the nat-test-1 instance (use “ssh -A”) and check that we can’t access the Internet from nat-test-1 or backend-1.

Test Internet access

Let’s create our NAT gateway. We need to select the VPC network, the region and create a new Cloud Router (or select an existing one).

In the NAT mapping section, we can select which subnets to map to the NAT gateway. By default, all subnets are allowed to use the NAT gateway but we are going to restrict them to only subnet-us-central-dmz:

Next section is NAT IP addresses. We can select two options:

• Automatic (recommended for scalability): allows auto-allocation of public IPs to the NAT gateway as the number of instances increases or decreases.
• Manual (my recommended choice): manually add one or more NAT IPs (1 or more IPs in multiple of 2) to be used by the NAT gateway. These IPs have to be reserved static IP addresses. In automatic mode, we don’t have full control of IPs and they can change over time and it’s impossible to maintain in a firewall. The main drawback is that we have to monitor that we have enough public IPs associated with the service (we will see later how to monitor).

NAT IP addresses

At the end of the page, we can review the Minimum port and timeouts.

One critical aspect of the set up is the Minimum ports per VM instance. By default, a VM instance has 64 ports, which means it can open 64 connections to the same destination IP and port (it can open another 64 TCP and 64 UDP connections to a different destination IP address and port). An instance can open up to 64000 connections across all protocols. Every Cloud NAT IP address has 65536 ports available for TCP and another 65536 for UDP. Of these, the first 1024 well-known ports are not used by Cloud NAT. So we have 64512 available ports per NAT IP address. Therefore, up to 1008 VMs (65536/64) can be supported by a single NAT IP address. If we add a second NAT IP address we can have up to 2016 VMs, etc (remember: if we setup more than 1 NAT IPs, we need an even number). If we add a second IP, every instance reserves half of the ports with one reserved IP and the other half with the other.

If we are using Google Kubernetes Engine (GKE), the per-VM number of ports is the number of ports available to all containers on that VM.

Speaking of timeouts, the default configuration is:

• UDP Mapping Idle Timeout: 30s
• ICMP Mapping Idle Timeout: 30s
• TCP Established Connection Idle Timeout: 1200s
• TCP Transitory Connection Idle Timeout: 30s

It has to be taken into consideration that NAT mappings are checked every 30s for expiration and configuration change. Even if a connection timeout value of 5 seconds is used, the connection may not be available for up to 30s in the worst case, and 15 seconds in the average case.

After we have created our NAT gateway, we can check if the configuration is valid:

It’s time to test our download speed. We are going to use the tool clispeedtest-cli (see how to install it in Github). After we install the tool, we are going to make a test from our bastion host and our natted instance to the closest server

And one final test: we are going to limit the Minimum ports per VM instance to see how the service is affected. Our first step is to decrease the number of ports per VM instance to 8.

Note: We can change the Minimum ports per VM instance at any time; but, if we decrease the number, we need to stop and start the VM instance (a reboot doesn't work) to apply the change of the number of ports reserved by the VM. We checked with Google support because this is currently undocumented and they confirmed it's the expected behavior.

After our instance is on-line again, we are going to repeat our test and we can see that the download speed is lower than before (1.14 Mbit/s). This is because the speed test opens several connections to the origin server at the same time, while having a lower number of ports available.

In order to see the status of the service, we could :

• NAT IP:port-ranges: we can review the port assigned to each VM instance

gcloud beta compute routers get-nat-mapping-info <nat_router>  - region <region>

• NAT status: we have to pay special attention to minExtraNatIpsNeeded to see if we need to add more IPs in case we setup NAT IPs addresses to Manual

gcloud beta compute routers get-status <nat_router> - region <region>

Finally, if you’re wondering about pricing, Cloud NAT is provided for free during Beta stage. After that, pricing (which will be per region) will be based on the following:

• an hourly price for the NAT gateway
• a per/GB cost for ingress and egress traffic processed by the gateway

If you have a problem, if no one else can help, and if you can find them, maybe you can hire the Bluekiri Team.