Secure access to your VMs in Google Cloud

One of the requirements for secure access remote machines are an easy management of users to VMs access matrix and reduced solution and operational costs. For VM fleets hosted on Google Cloud it is easy to do. In this post I build an example of the working solution based on OS-Login and IAP Tunneling and review a few important focus points about the technology and best practices.

How it is usually done.

Majority of the existing solutions for Data Centers and Cloud providers use one of three constructs:

• SSH / RDP encrypted connection to destination host
• SSH / RDP encrypted connection to a Bastion host that serves as a jump location to destination host
• Virtual desktop vendor solution that a user connects to access the destination host

These constructs have few drawbacks. SSH based solutions require expensive (in money and effort) SSH key management and have a problem to bind the SSH key that was used to a principal account of the user who performed the operation. RDP based solutions usually require establishing a VPN connection that has high operational costs and hard to use for end users. And vendor solutions are never cheap.

How it is done in Google Cloud.

Google Cloud lets users connect to VMs that are not exposed to the internet without a need for VPN connection using IAP Tunneling. This allows eliminating the need for VPN connections while keeping the fleet of VMs unavailable for external attacks. With Google Cloud you can manage SSH keys on a per-user basis implicitly without exposing them to end users thus mitigating risks related to the key management or lost key. It also allows you to attribute the access to VM instances to principal users. This functionality is called OS Login and can be enabled for a single VM instance or to all VM instances in the project or organization. The use of OS Login leverages the Google Cloud IAM service and simplifies the users to VMs management matrix. The use of IAP Tunneling and OS Login does not carry additional costs.

Like everything this solution has limitations. The OS Login only works for SSH connectivity. It cannot be used with RDP. It should not be a problem for modern versions of Windows since all of them come with built-in SSH support. However, it is impossible to have a connection with GUI. In this case you can leverage other Google Cloud solutions such as RDP or Virtual Desktop.

Less talk, more code.

There are plenty of articles with code samples that demonstrate IAP Tunneling and OS Login. The example below differs only by using most recent Cloud SDK commands and, probably, being minimalistic.

To run it you have to have the roles/editor role or similar set of permissions on a GCP project with a default VPC and to have Cloud SDK installed on your client machine. The instructions are provided for a client machine with Linux OS. It should be easy to adapt them to run on Windows.

1. Setup configuration environment by defining a zone, project id and VM instance name:

# VM_NAME=sample-destination-host; \
ZONE=us-central1-a; \
PROJECT_ID=<your project id>; \
USER_ID=$(gcloud config list account --format "value(core.account)")

If you want to use the example in order to grant another Google account to connect to the destination host, place that user’s email address as a value for USER_ID.

2. Authenticate with Cloud SDK. The same user will be used for configuring the example and connecting to the destination host:

# gcloud auth login

3. Ensure that all relevant APIs are enabled:

# gcloud services enable compute.googleapi.com \
    iap.googleapis.com \
    --project $PROJECT_ID

4. Create a sample destination host:

# gcloud compute instances create $VM_NAME \
    --network default \
    --metadata enable-oslogin=TRUE \
    --role roles/compute.osAdminLogin \
    --zone $ZONE \
    --project $PROJECT_ID

The currently authorized user is granted the roles/compute.osAdminLogin role on the VM instance. It is possible to grant the roles/compute.osLogin role which does grant administrator permissions on the VM instance.

6. Allow the user to create SSH (port 22) IAP Tunneling in the project:

# gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member user:$USER_ID \
    --role roles/iap.tunnelResourceAccessor \
    --condition \
      'title=allow-ssh-only,expression=destination.port==22'

At this moment Cloud SDK does not support defining IAM policy for a single VM instance. It can be done only in the Cloud Console. The example defines the policy for the user on the project.

7. Configure firewall rules to allow SSH connection from IAP to the default VPC network.

# gcloud compute firewall-rules create allow-ssh-ingress-from-iap \
    --network default \
    --direction INGRESS \
    --action allow \
    --rules tcp:22 \
    --source-ranges 35.235.240.0/20 \
    --project $PROJECT_ID

It is possible now to connect to the instance using the current user credentials:

# gcloud compute ssh --tunnel-through-iap \
    “projects/$PROJECT_ID/zones/$ZONE/instances/$VM_NAME”

If you want to use a different Google account to connect to the destination host, you will have to re-authenticate for Cloud SDK before executing the above command.

In-depth review of what we did and best practices.

When you call “gcloud compute ssh” the following happens:

1. Cloud SDK from the client machine creates TCP Tunnel via IAP to port 22 on the Destination VM.

2. IAP verifies that the user (current active credentials in the Cloud SDK on the client) is authorized to establish IAP tunneling to the Destination VM.

3. SSH connection is established to the Destination VM.

4. The OS-Login agent on the Destination VM verifies that the user is authorized to login. In addition, the user is verified having the Service Account User role on the VM if the VM has an attached service account.

The IAP tunneling permissions can be granted on the project or VM instance level. The firewall rules can be defined in the organizational, folder or project level using hierarchical firewall policies. Similarly, you can enforce OS-Login on the instance or project level or to define it as an organizational policy to enforce it for the whole VM fleet of your organization.

Granting IAP tunneling and OS Login permissions on the project or organizational levels the fine grained access control to a particular VM is reached using Google service accounts. Users cannot login to a VM instance if they are not granted the Service Account User role to the service account that is attached to that instance. Unless it is explicitly configured at creation time VM instances will have a default Compute Engine service account attached to them. The account is automatically provisioned for each project and looks like 123456789-compute@developer.gserviceaccount.com. It is considered a bad practice to use this account. The recommended practice is to provision dedicated service accounts for VM instances.

Wrap up…

• Using OS Login eliminates the burden of managing SSH keys and mitigates potential security issues with lost keys. However, it cannot be used with connection methods such as RDP.
• Using IAP Tunneling allows to keep the entire VM fleet in the private network address space and avoid exposing VMs for access from the internet.
• Following best practices in using dedicated service accounts and organizational policies provides out of the box solution for managing secure access for users in your organization to the whole VM fleet hosted in Google Cloud.
• The solution does not incur any additional costs.
• All operations are automatically logged into Cloud Logging for further auditing including identity of the principals that initiated the connection to VMs. Alerting policies can be defined to notify about unexpected or abnormal connections.