Google Cloud HA VPN to Compute Engine VM (NVA) config lab with VyOS

In January 2024 Google Cloud announced the GA of the Cloud HA VPN support of compute Instances with public IPs. This is a good use case to switch over from classic VPN to HA VPN and connect to NVA devices running on compute Instance within Google Cloud. You can read more about the feature here.

It’s always fun to lab up things and so to do this test I utilize:

# 1 - VyOS from Google Cloud marketplace, running in one project.
# 2 - In another project, I configure the HA VPN with the option Compute Engine VMs with external IP addresses. Now let’s jump to the setup.

Prerequisites:

• You have a project created (I used two projects but you can use one project with two VPCs)
• You have permissions to deploy cloud VPN, compute, Firewall in your environment.
• You have access to marketplace (note this will incur a cost)

Design Single VM

Project 1 — VyOS VM appliance.

# 1 — Create a project for the virtual appliance. Note you can also use one project and just create two VPCs to test the connectivity. (Optional skip if you already have your test project and network.)

Open cloud shell and configure as follows.
p.s change YOUR-PROJECT-ID for the name of your project.

gcloud config list project
gcloud config set project YOUR-PROJECT-ID
projectid=YOUR-PROJECT-ID
echo $projectid

Create VPC

gcloud compute networks create nva-network --project=$projectid \
--subnet-mode=custom \
--mtu=1460 \
--bgp-routing-mode=regional

Create Subnet

gcloud compute networks subnets create nva-subnet \
--project=$projectid --range=10.0.211.0/24 \
--stack-type=IPV4_ONLY --network=nva-network \
--region=europe-west1

Create Default Firewall rules

gcloud compute firewall-rules create nva-network-allow-custom \
--project=$projectid \
--network=projects/$projectid/global/networks/nva-network \
--description=Allows\ connection\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ custom\ protocols. \
--direction=INGRESS --priority=65534 \
--source-ranges=10.0.0.0/16 \
--action=ALLOW \
--rules=all

gcloud compute firewall-rules create nva-network-allow-icmp \
--project=$projectid \
--network=projects/$projectid/global/networks/nva-network \
--description=Allows\ ICMP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW \
--rules=icmp

gcloud compute firewall-rules create nva-network-allow-rdp \
--project=$projectid \
--network=projects/$projectid/global/networks/nva-network \
--description=Allows\ RDP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 3389. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW --rules=tcp:3389

gcloud compute firewall-rules create nva-network-allow-ssh \
--project=$projectid \
--network=projects/$projectid/global/networks/nva-network \
--description=Allows\ TCP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 22. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 --action=ALLOW \
--rules=tcp:22

Create VyOS appliance.
The first thing you need to do is create a key and add to the metadata section of Google Cloud. If you don’t do this you will not be able to ssh into the appliance. This is documented here https://docs.vyos.io/en/equuleus/installation/cloud/gcp.html

• Create ssh key in cloud shell. When prompted add a passphrase I used something simple like vyos

ssh-keygen -t rsa -f ~/.ssh/vyos_gcp -C "vyos@mypc"

• Open cloud shell in editor mode and navigate to the /home/username/.ssh/ folder and you will see the files created with the vyos at the start. Open the public key file and copy the content.
• Navigate to the menu Metadata. Choose SSH Keys and click edit and then select Add Item
• Copy the key content here and save.

Install VyOS from marketplace

• Go to marketplace search for VyOS and select the options, this will prompt you with what to do.

• Ensure the correct project is selected, for the network in this example we selected the nva-network
• For the firewall rule we used the 0.0.0.0/0 network for port ICMP,22, UDP 500, 4500
• Ensure IP forwarding is selected
• Deploy. You will see the deployment management area open showing the deployment and status. When completed you will have a device.

vyos instance deployment tracking

• Navigate to the VM Instances page and copy the External IP address of the VM

vm instance

Connect to VyOS device
Connect from cloud shell
p.s change ADD-VM-EXTERNAL-IP to the external IP of the VyOS VM.
Note it will also prompt you for the passphrase

ssh -i ~/.ssh/vyos_gcp vyos@ADD-VM-EXTERNAL-IP

Enter config mode type configure

Set the ipsec interface and vti interfaces. We are going to use 169.254.150.250/30 and 169.254.140.178/30 addresses

set vpn ipsec ipsec-interfaces interface eth0 

set interfaces vti vti11 address '169.254.150.250/30' 
set interfaces vti vti11 description 'gcp-ha-peer-1'
set interfaces vti vti11 mtu '1460'
set interfaces vti vti12 address '169.254.140.178/30'
set interfaces vti vti12 description 'gcp-ha-peer-2'
set interfaces vti vti12 mtu '1460'

Set the IPSEC settings

set vpn ipsec esp-group ESP-TO-HA-GCP lifetime '3600'
set vpn ipsec esp-group ESP-TO-HA-GCP pfs 'dh-group14'
set vpn ipsec esp-group ESP-TO-HA-GCP proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-TO-HA-GCP proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-TO-HA-GCP dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-TO-HA-GCP dead-peer-detection interval '20'
set vpn ipsec ike-group IKE-TO-HA-GCP dead-peer-detection timeout '80'
set vpn ipsec ike-group IKE-TO-HA-GCP key-exchange 'ikev2'
set vpn ipsec ike-group IKE-TO-HA-GCP lifetime '28800'
set vpn ipsec ike-group IKE-TO-HA-GCP proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-TO-HA-GCP proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-TO-HA-GCP proposal 1 hash 'sha256'

We need additional info like the external IP addresses of the Google Cloud HA gateway to complete the config so let’s jump over to the other Google Cloud VPC to set up the HA VPN gateway

Project 2— HA VPN gateway

I’m using two projects. However you can use one project with different VPCs.
# 1 — Create a Project (optional)
# 2 — Create a VPC with subnet (region of your choice)
# 3 — Create Cloud Router
# 4 — Create VM instance you can use for testing
# 5— Create HA VPN
# 6— Create tunnels and BGP
# 7— Test connectivity from test vm

p.s change YOUR-PROJECT-ID for the name of your project (used to create the HA VPN) Skip is using same project as before.

gcloud config list project
gcloud config set project YOUR-PROJECT-ID
projectid2=YOUR-PROJECT-ID
echo $projectid2

Create VPC and subnets. (This is for HA VPN)

gcloud compute networks create ha-vpn-net --project=$projectid2 \
--subnet-mode=custom \
--mtu=1460 \
--bgp-routing-mode=regional

gcloud compute networks subnets create vm-subnet \
--project=$projectid2 --range=192.168.20.0/24 \
--stack-type=IPV4_ONLY --network=ha-vpn-net \
--region=us-east1

Create Default Firewall rules

gcloud compute firewall-rules create ha-vpn-net-allow-custom \
--project=$projectid2 \
--network=projects/$projectid2/global/networks/ha-vpn-net \
--description=Allows\ connection\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ custom\ protocols. \
--direction=INGRESS --priority=65534 \
--source-ranges=10.0.0.0/16 \
--action=ALLOW \
--rules=all

gcloud compute firewall-rules create ha-vpn-net-allow-icmp \
--project=$projectid2 \
--network=projects/$projectid2/global/networks/ha-vpn-net \
--description=Allows\ ICMP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW \
--rules=icmp

gcloud compute firewall-rules create ha-vpn-net-allow-rdp \
--project=$projectid2 \
--network=projects/$projectid2/global/networks/ha-vpn-net \
--description=Allows\ RDP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 3389. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW --rules=tcp:3389

gcloud compute firewall-rules create ha-vpn-net-allow-ssh \
--project=$projectid2 \
--network=projects/$projectid2/global/networks/ha-vpn-net \
--description=Allows\ TCP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 22. \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 --action=ALLOW \
--rules=tcp:22

Create Cloud Router

gcloud compute routers create ha-vpn-cr \
--project=$projectid2 \
--region=us-east1 \
--network=ha-vpn-net \
--asn=64512

Create test VM for connectivity test

gcloud compute instances create vpn-test-vm \
--project=$projectid2 \
--zone=us-east1-b \
--machine-type=e2-medium --network-interface=stack-type=IPV4_ONLY,subnet=vm-subnet,no-address,network=ha-vpn-net 

Now we create the HA VPN
1 — gateway, peer gateway, tunnel
2 — BGP sessions

Create gateway

gcloud compute vpn-gateways create vpn-to-vm-gw \
   --network=ha-vpn-net \
   --region=us-east1

Inside the console view record the IP of the interfaces. You will need these to complete the setup on the VyOS vm.

Create Peer gateway

p.s change PUT-VYOS-EXTERNAL-IP to the External IP address of the VyOS VM.

gcloud compute external-vpn-gateways create gw-to-vyos \
     --interfaces 0=PUT-VYOS-EXTERNAL-IP

Create VPN Tunnels

gcloud compute vpn-tunnels create vpn-to-vm-tun0 \
    --peer-external-gateway gw-to-vyos \
    --peer-external-gateway-interface  0 \
    --region us-east1 \
    --ike-version 2 \
    --shared-secret vyos \
    --router ha-vpn-cr \
    --vpn-gateway vpn-to-vm-gw \
    --interface 0

gcloud compute vpn-tunnels create vpn-to-vm-tun1 \
    --peer-external-gateway gw-to-vyos \
    --peer-external-gateway-interface  0 \
    --region us-east1 \
    --ike-version 2 \
    --shared-secret vyos \
    --router ha-vpn-cr \
    --vpn-gateway vpn-to-vm-gw \
    --interface 1

Configure BGP on Tunnels

gcloud compute routers add-interface ha-vpn-cr \
    --interface-name session-t0 \
    --ip-address 169.254.150.249 \
    --mask-length 30 \
    --vpn-tunnel vpn-to-vm-tun0 \
    --region us-east1

gcloud compute routers add-bgp-peer ha-vpn-cr \
    --peer-name bgp-vyos-tunnel0 \
    --interface session-t0 \
    --peer-ip-address 169.254.150.250 \
    --peer-asn 64513 \
    --region=us-east1

gcloud compute routers add-interface ha-vpn-cr \
    --interface-name session-t1 \
    --ip-address 169.254.140.177 \
    --mask-length 30 \
    --vpn-tunnel vpn-to-vm-tun1 \
    --region us-east1

gcloud compute routers add-bgp-peer ha-vpn-cr \
    --peer-name bgp-vyos-tunnel1 \
    --interface session-t1 \
    --peer-ip-address 169.254.140.178 \
    --peer-asn 64513 \
    --region=us-east1

At this point the connection should be waiting so we need to complete the configuration on the vyos instance.

awaiting connection

Switch to Project 1 — VyOS VM appliance.

Switch back to the VyOS project. If disconnected reconnect to the vyos device by ssh in cloud s shell. Make sure your cloud shell session in linked to the correct project if you are using different projects.

In this case the IP addresses to note and change in the config are
- Local-IP — (Put the local private IP of your VyOS vm)
- VyOS-External-IP — (Put the VyOS external IP)
- GCP-External-IP-Tun0 (Put gcp ha vpn external IP Tun0)
- GCP-External-IP-Tun1 (Put gcp ha vpn external IP Tun1)

Configs Tunnel on VyOS (for that that match the above input your devices information)

set vpn ipsec site-to-site peer GCP-External-IP-Tun0 authentication id 'VyOS-External-IP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 authentication pre-shared-secret vyos
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 authentication remote-id 'GCP-External-IP-Tun0'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 connection-type 'initiate'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 ike-group 'IKE-TO-HA-GCP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 local-address 'Local-IP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 vti bind 'vti11'
set vpn ipsec site-to-site peer GCP-External-IP-Tun0 vti esp-group 'ESP-TO-HA-GCP'

set vpn ipsec site-to-site peer GCP-External-IP-Tun1 authentication id 'VyOS-External-IP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 authentication pre-shared-secret vyos
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 authentication remote-id 'GCP-External-IP-Tun1'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 connection-type 'initiate'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 ike-group 'IKE-TO-HA-GCP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 local-address 'Local-IP'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 vti bind 'vti12'
set vpn ipsec site-to-site peer GCP-External-IP-Tun1 vti esp-group 'ESP-TO-HA-GCP'

Important reminder (you must swap out these values in the above config with the correct IPs for it to work)
- Local-IP — (Put the local IP of your VyOS vm)
- VyOS-External-IP — (Put the VyOS external IP)
- GCP-External-IP-Tun0 (Put gcp ha vpn external IP Tun0)
- GCP-External-IP-Tun1 (Put gcp ha vpn external IP Tun1)
Sample image with IP below

Type commit to save

Once all the configs are accepted the tunnel should be up. from config mode type

run show vpn ike sa

Configure BGP on VyOS

set protocols bgp 64513 neighbor 169.254.150.249 remote-as '64512'
set protocols bgp 64513 neighbor 169.254.150.249 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 64513 neighbor 169.254.150.249 ebgp-multihop '10'
set protocols bgp 64513 neighbor 169.254.150.249 remote-as '64512'
set protocols bgp 64513 neighbor 169.254.150.249 timers holdtime '60'
set protocols bgp 64513 neighbor 169.254.150.249 timers keepalive '20'

set protocols bgp 64513 neighbor 169.254.140.177 remote-as '64512'
set protocols bgp 64513 neighbor 169.254.140.177 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 64513 neighbor 169.254.140.177 ebgp-multihop '10'
set protocols bgp 64513 neighbor 169.254.140.177 remote-as '64512'
set protocols bgp 64513 neighbor 169.254.140.177 timers holdtime '60'
set protocols bgp 64513 neighbor 169.254.140.177 timers keepalive '20'

set protocols bgp 64513 address-family ipv4-unicast network 10.0.211.0/24

type commit to save

It may take a minute or two for bgp to come up verify from config prompt

run show bgp summary 

run show ip route bgp

Connectivity test

Do a ping test from vyos device to IP address of test VM connected to VPN gateway. In this case the VM IP adrress is 192.168.20.4 we have to ping fron the local IP as the source which is 10.0.211.2

run ping 192.168.20.4 source-address 10.0.211.2

On the other side we can ping from the test VM to the local interface of the device.
SSH into the VM and type

ping -c 4 10.0.211.2

Congrats !!! Test and configurations completed.

Make sure and delete the following after your testing if you follow this labs.
- VyOS deployment (go to deployment manager you will see this deployment select it and choose delete all include dependencies.)
- Delete the VPN (gateway, peer gateway, tunnels)
- Delete the test instance

VPN Documentation — https://cloud.google.com/network-connectivity/docs/vpn/concepts/topologies

Follow me on LinkedIn.

I’ll be in touch