Connecting Securely to Google Compute Engine VMs without a Public IP or VPN

In this article I will show you how to establish secure RDP, ssh and VNC connections to VMs on GCE that don’t have a public IP or VPN connectivity.

The proposed proxy based configuration reduces the attack surface of GCE VMs as they don’t have to be exposed to the outside world. For Windows nodes the Remote Desktop Protocol (RDP) is one of the most common attack vectors used by malicious actors. For Linux nodes brute force ssh attacks are a common method to gain unauthorized access. The configuration also allows for controlling remote access at the granularity of the individual VM. This addresses a drawback of VPNs that grant access to all systems behind a VPN under the assumption that the network is an effective security perimeter.

While the focus is on RDP and ssh protocols, the described tunneling method works for all TCP ports.

Google Cloud Platform’s (GCP) Identity-Aware Proxy (IAP)

For TCP connections, GCP’s Identity-Aware Proxy (IAP) enables access to GCE VMs from the internet through their private IP. When an attempt is made to establish an HTTPS encrypted tunnel to the proxy, the proxy performs an authentication and authorization check. After successful authentication and authorization, traffic from the client is forwarded on Google’s internal network to the VM instance by the proxy. In other words, the proxy acts as a gatekeeper for incoming traffic that is exposed to the outside world while the VM only has to allow for connections from the proxy on the Google internal network.

We will first describe the basic IAP configuration. Then we will cover

• Linux to Linux connections with ssh
• Linux to Linux connections with VNC
• Windows to Windows connections with RDP
• Linux to Windows connections with RDP

After discussing the basic configuration and setup we will discuss how you can use GCP’s Access Context Manager in conjunction with IAM conditions for access control. Access Context Manager grants access levels based on request attributes as for example the IP of the client that attempts to establish a connection. Lastly we will walk through a basic Cloud NAT configuration that enables outbound connectivity for your VM.

Tunnel Configuration

To configure the IAP tunnel I am assuming that you have the following

• A GCP project you have ownership of with a running GCE VM and for Windows VMs, login and password — make sure that you disable the assignment of a public IP when you create the VM — for Windows nodes make sure that you don’t forget to specify username and password after the VM has been created
• The Google Cloud SDK installed on the system you want to connect from

Before we start, please make sure that your project and user account are set correctly. You can verify with ‘gcloud config list’ that your settings are correct and if necessary change the account you are logged into with ‘gcloud auth login’. The project can be set with ‘gcloud config set core/project’. If you are behind a corporate proxy you may have to whitelist the domain ‘tunnel.cloudproxy.app’ used by IAP for TCP. For the most part, I will be describing the configuration through the CLI but most tasks can also be accomplished through the GCP console UI.

First we need to create a firewall rule that enables traffic from the IAP to the VM. IAP uses the range 35.235.240.0/20 as a source address for forwarding traffic. We allow for ports 22 (ssh), 3389 (rdp) and 5900 (vnc). You can pick and choose the port number(s) depending on the protocol you need.

gcloud compute firewall-rules create allow-ingress-from-iap \
--direction=INGRESS \
--action=allow \
--rules=tcp:3389,tcp:22,tcp:5901 \
--source-ranges=35.235.240.0/20

This command allows for connections from the proxy to all nodes in your project. If you don’t want to open up the firewall for all VMs in your project you can attach a tag to the VMs you want the rule to apply to and specify that tag with ‘ — target-tags=TAG’ as an additional option.

Now we need to add an IAM policy that allows users to establish tunneled connections. We grant the IAM role iap.tunnelResourceAccessor to all users and/or user groups that are authorized to use the tunnel. In this example this role is set at the project level. This allows users to establish connections to all VMs in the project to which the above firewall rules apply. We will see later how we can be more selective and only enable tunneling to specific instances without changing the firewall configuration.

gcloud projects add-iam-policy-binding iap-access-test \
--member=user:testuser@example.com \
--role=roles/iap.tunnelResourceAccessor

The user also needs to have the compute.viewer role

gcloud projects add-iam-policy-binding iap-access-test \
--member=user:testuser@example.com \
--role=roles/compute.viewer

Connecting with SSH: Linux-Linux on GCE

If you want to connect to a Linux node via the VNC protocol you establish the tunnel for port 5901 which is the port number for the first VNC session hosted by iap-test-ubuntu

gcloud compute start-iap-tunnel iap-test-ubuntu 5901 \
--local-host-port=localhost:5901

Alternatively you could also use port forwarding over ssh

gcloud compute ssh iap-test-ubuntu --project iap-access-test \
--zone us-central1-a --ssh-flag "-L 5901:localhost:5901"

After the tunnel is established you can point the VNC client, for example Remmina or the VNC Viewer to port 5901 on the local host to connect to your VM instance through your tunnel.

If you want ‘click-of-a-button’ connectivity with Remmina you can create a short launch script. Before writing the script you need to create a configuration profile for the connection. Open the Remmina Desktop Client and add a new connection profile clicking on ‘+’ in the upper left. All we need to specify is the type of connection (Remmina VNC Plugin), the profile name and localhost:5901 as the server:port. The local host is the starting point for the tunnel to the Windows VM through IAP. ‘Save’ to make sure the connection settings are saved.

The new profile is saved in ~/.local/share/remmina. Now you can create a small startup script and place it on the Desktop for instant access.

#!/bin/bash
gcloud config set core/project iap-access-test
gcloud compute start-iap-tunnel iap-test-ubuntu 5901 \
--local-host-port=localhost:5901 --zone=us-central1-a &
sleep 2
remmina -c /home/adittmer/.local/share/remmina/1604188073948.remmina

We will not describe here how to install a Desktop like Gnome on a GCE instance. We are also not covering the server side VNC configuration. This article is a great starting point if you add the following to the ~/.vnc/xstartup file suggested in the article.

gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &

Connecting with RDP: Linux-Windows on GCE

For connecting to Windows VM instances I also used the Remmina client. The only difference from the VNC client example is obviously the port number as well as a few minor differences in the Remmina UI. For the tunnel setup we run

gcloud compute start-iap-tunnel iap-test-windows 3389 \
--local-host-port=localhost:3389

When you create a Remmina profile as per the instructions above you need to specify RDP as a protocol. For ‘Server’ you just enter ‘localhost’ without a port number. You can use the script above but need to change the port number 5901 to 3389, modify the VM’s hostname and reference the right profile for the RDP connection.

Connecting with RDP: Windows-Windows on GCE

For RDP connections from Windows clients IAP Desktop is a great and easy solution. The installation wizard initiates an Oauth flow to authenticate against your Google user account. After initialization it shows the available VMs. Double-clicking on the VM instance creates a tunnel and an RDP connection.

Killing IAP Tunnel Connections

Connections launched from your shell with gcloud can be killed through CTRL-C. Connections running in the background can be killed with

kill $(lsof -ti tcp:<port number>)

Access Levels

If you want to be more restrictive about devices that can establish IAP tunnels you can use access levels in conjunction with IAM conditions. Access levels are granted by IAP based on attributes of a request. IAP can for example assign an access level to a request for initiating a tunneled connection based on the source IP of the request. If an access level is assigned and matches the level specified in the IAM condition for the ‘iap.tunnelResourceAccessor’ role for the requesting user the tunnel is established.

Access levels are defined through the Access Context Manager and are a construct at the organization level. This means that all access levels are visible in all projects. Access Context Manager supports basic and custom access levels. Basic access levels only allow for conditions that can be ‘AND’ or ‘OR’ combined. Custom levels allow for more complex expressions and support a Common Expression Language (CEL) for filtering based on request type and attribute. Custom access levels are only available as part of a paid enterprise security subscription. For this example, we will define a basic access level.

The users that are authorized to create access levels need to have the resource manager organization editor role as well as the access context manager policy admin role. These roles can be assigned by the organization admin (an individual with the role roles/resourcemanager.organizationAdmin). The organization ID can be obtained with ‘gcloud organizations list’.

gcloud organizations add-iam-policy-binding 505850696945 \
--member="user:newuser@example.com" \
--role roles/editor
gcloud organizations add-iam-policy-binding 505850696945 \
--member="user:newuser@example.com" \
--role roles/accesscontextmanager.policyAdmin

Before we continue and explore how we can set restrictions for the creation of tunnels we need to revoke the broad access we granted earlier with ‘gcloud projects add-iam-policy-binding’, that granted the iap.tunnelResourceAccessor role to all VMs without any conditions other than that the user is authenticated.

gcloud projects remove-iam-policy-binding iap-access-test \
--member testuser@example.com \
--role "roles/iap.tunnelResourceAccessor”

Note that project owners will always have unrestricted tunnel access.

Restricting Access by Source IP and Geography

My use case is that I travel frequently to Canada and want to create a policy that allows tunnel creation only from specific IPs that are either in Canada or the US. I create the YAML file below that assigns the access level ‘secureaccess’ to requests from the specified (redacted) IP addresses in the respective geographies. The two IP addresses and geos are ‘OR’ combined, IP ranges and regions are ‘AND’ combined per default. This can be changed with the — combine-function argument of the ‘gcloud access-context-manager levels create’ command.

- ipSubnetworks:
     - 73.21.45.151/32
     - 201.23.121.234/32
- regions:
     - US
     - CA

We create the access level

gcloud access-context-manager levels create secureaccess \
--title myaccesspolicy \
--basic-level-spec access-canada-us.yaml

In the GCP console you can create and edit the access levels if you go to ‘Security | Access Context Manager’

Now we create an IAM condition that grants the ‘tunnel user’ (iap.tunnelResourceAccessor) role to the user if the request is at the access level ‘secureaccess’. The ‘tunnel user’ role with IAM conditions has to be set for the resource it applies to. Unfortunately there is currently no way to accomplish this through the gcloud CLI so we use the console.

1. Go to ‘Security | Identity-Aware Proxy’ and select your project.
2. Select ‘SSH AND TCP RESOURCES’ and you will see a resource hierarchy for tunnel resources — with the checkboxes you can specify the scope of the IAM condition you are about to create — you can apply the IAM condition to all tunnels in a project or zone or to a specific VM instance.
3. Click on ‘Add Member’ and enter the identity of the user (testuser@example.com),
4. Select the role IAP-secured Tunnel and
5. Add the IAM condition with the Condition Builder or Editor (see screenshot below)
6. ‘Save’

The inheritance hierarchy from project to zone to VM that prevents you from overriding and blocking policies set for a higher level in the hierarchy. You can, for example, not remove or modify an IAM condition that was set for the project for one VM in the project.

If you need to set an IAM condition for iap.tunnelResourceAccessor on a resource from the command line, you can use the REST API for IAP. In contrast to the UI where we identified the access level through its title in the drop down, the API uses the long access level name. You can obtain the long name with ‘gcloud access-context-manager levels list — format list’

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "content-type: application/json" -X POST https://iap.googleapis.com/v1beta1/projects/307931860343/iap_tunnel:setIamPolicy -d '{"policy":{"bindings":[{"role":"roles/iap.tunnelResourceAccessor","members":["user:testuser@example.com"],"condition": {"description": "Test","expression": "\"accessPolicies/78305086638/accessLevels/secureaccess\" in request.auth.access_levels","title": "restricted access"}}]}}'

If you want to assign the policy at the level of a specific instances you can do that by changing the URL of the request to the URL for the VM instance you want to enable access for,

https://iap.googleapis.com/v1beta1/projects/<project_id>/iap_tunnel/zones/<vm zone>/instances/<instance-name>:setIamPolicy

for example

https://iap.googleapis.com/v1beta1/projects/185772708261/iap_tunnel/zones/us-central1-a/instances/my-instance:setIamPolicy

If you want to apply the condition to a zone the URL format is

https://iap.googleapis.com/v1beta1/projects/<project_id>/iap_tunnel/zones/<vm zone>

Note that the API call overwrites whatever policy is set for the respective resource with the exception of policies inherited from the higher level of the resource hierarchy.

Enabling Outbound Connectivity for your GCE VM with GCP Cloud NAT

If you require outbound access from your GCE VM to the internet and your VM behind the proxy you can use GCP’s Cloud NAT. Cloud NAT provides source network address translation (SNAT) for VMs without external IP addresses. Cloud NAT also provides destination network address translation (DNAT) for established inbound response packets.

We start by creating a Cloud Router instance

gcloud compute routers create mycloudrouter \
--project=iap-access-test \
--region=us-central1 --network=default

We set the NAT configuration for the Cloud Router

gcloud compute routers nats create mynatconfig \
--router=mycloudrouter --nat-all-subnet-ip-ranges --enable-logging \
--auto-allocate-nat-external-ips

The flag auto-allocate-nat-external-ips ensures that SNAT addresses are assigned automatically. nat-all-subnet-ip-ranges allows NATing of all IP ranges of all subnetworks in the region, including primary and secondary ranges. These instructions just cover a relatively simple configuration. For more information please consult the GCP documentation.

Summary

We covered quite a bit of ground in this article. We first discussed GCP’s IAP and then created a basic configuration that allows VM access from anywhere through the proxy forwarding traffic to your VMs. Then we looked into access control through access levels and went through an example that only establishes tunnels when requests come from a specified IP and geography. We also saw how access can be restricted to VMs in specific zones as well as to specific VMs. Lastly we went through a quick configuration of GCP Cloud NAT in case your VM needs outbound access.