Connecting to Filestore from a different VPC

Filestore — a Google Cloud product for managed NFS — allows you to mount shared disks to VMs, containers, kubernetes services — anything with network connectivity and the ability to connect to NFS. Managed NFS sounds good, but it’s not a serverless solution — nor does it live in your VPC — which creates some interesting debugging opportunities…

The Back Story

We were working on a project to move a GKE-based environment to a shiny new foundation, with more security and access to the bleeding edge of innovation. Data migration was going to happen after the workloads were moved, so the decision was made to use the existing Filestore instance in the old project in the meantime. Let’s call this old project project1 , and the shiny new one project2. Yes, very boring!

You might find yourself in a similar situation if you need to access Filestore from on-prem, another cloud, or simply a different project and VPC.

The Challenge

Filestore, like Cloud SQL, does not deploy to your VPC. Instead, the instance exists in a Google-managed VPC, a peering is established with your VPC, and you get the IP address of the file share — you use that to mount it. This peering generates a route in your network to point the Filestore IP range to the VPC peering, like so:

Filestore, like Cloud SQL, is deployed into a Google-managed VPC, and a peering is established with your VPC

We can observe these connections in the console (Peerings, and Routes), or we can run some gcloud commands:

➜ gcloud compute networks peerings list --project old-project
NAME                  NETWORK        PEER_PROJECT    PEER_NETWORK        STACK_TYPE  STATE_DETAILS
filestore-peer-123    project1-vpc   ab12cd34-ef     filestore-vpc-456   IPV4_ONLY   Connected.
➜ gcloud compute routes list --filter="nextHopPeering:'filestore-peer-123'" --project old-project
NAME                  NETWORK        DEST_RANGE      NEXT_HOP             PRIORITY
peering-route-123abc  project1-vpc   10.99.17.12/29  filestore-peer-123   0

VPC peering is non-transitive, meaning that if you peer 2 VPCs (VPC1 and VPC2), and try to access a Filestore instance in VPC1 from VPC2, you wouldn’t be able to. So what are the options? Well, essentially there is only one: VPN. You can create private connectivity between the two VPCs with a VPN, and then access Filestore.

The plan is to connect the VPC in project2 to the VPC in project1 via VPN

Side note — we did also consider Private Service Connect, but only because the Network page of the Filestore docs mentioned Private Services Access, and we confused the two. Together with Private Google Access, these form my current favourite cloud networking mindfuck ಡ_ಡ

Setting up the VPNs

Filestore docs have a dedicated page for accessing it via VPN, and another one for Known Issues — but the information there is pretty scarce. It pretty much says “Do the VPN 💫“ — with no detail. So let’s see how to set it all up.

We use Terraform, and, luckily, there is an official HA VPN module with examples on how to connect two VPCs in Google Cloud. This is handy, because setting up VPN connections involves quite a number of resources: VPN Gateways, routers, tunnels, exchanging IP addresses and secrets in the right places.. So we used the module. You’ll find our Terraform code at the end of this article.

Initially, we didn’t want the VPCs to learn each other’s routes, since this connectivity was only supposed to be between the new environment and the old Filestore. We decided to only advertise the Filestore peering route, and nothing in the opposite direction. This was a mistake!

You can advertise only the Filestore IP range from project1, but you do need the client side IP ranges to be known to the VPC in project1. It can be limited to only the subnet, or even a set of IP addresses, from which you will connect to this Filestore — for us, there was only one subnet in project2-vpc, with its IP range 10.1.0.0/18, so we went with the default advertisement mode for the router in project2.

Routes from the “client” subnets need to be advertised from the connecting project to the VPC in the project where Filestore is. In the opposite direction you can advertise only the Filestore IP range.

Terraform

Below is the Terraform configuration using the Google module for VPN:

# VPN on the "client" side - the shiny new project in our case
module "vpn_ha-1" {
  source           = "terraform-google-modules/vpn/google//modules/vpn_ha"
  version          = "~> 4.0"
  project_id       = "project2"
  region           = var.region
  network          = "https://www.googleapis.com/compute/v1/projects/project2/global/networks/project2-vpc"
  name             = "project2-to-project1"
  peer_gcp_gateway = module.vpn_ha-2.self_link # Link with the other side
  router_asn       = 64514
  router_advertise_config = {
    groups    = []
    ip_ranges = {}
    mode      = "DEFAULT"
  }
  tunnels = {
    remote-0 = {
      bgp_peer = {
        address = "169.254.1.1"
        asn     = 64513
      }
      bgp_session_range               = "169.254.1.2/30"
      ike_version                     = 2
      vpn_gateway_interface           = 0
      shared_secret                   = ""
    }
    remote-1 = {
      bgp_peer = {
        address = "169.254.2.1"
        asn     = 64513
      }
      bgp_session_range               = "169.254.2.2/30"
      ike_version                     = 2
      vpn_gateway_interface           = 1
      shared_secret                   = ""
    }
  }
}

# VPN on the "Filestore" side - the project hosting Filestore, in our case
module "vpn_ha_2" {
  source           = "terraform-google-modules/vpn/google//modules/vpn_ha"
  version          = "~> 4.0"
  project_id       = "project1"
  region           = var.region
  network          = "https://www.googleapis.com/compute/v1/projects/project1/global/networks/project1-vpc"
  name             = "project1-to-project2"
  router_asn       = 64513
  peer_gcp_gateway = module.vpn_ha-1.self_link # Link to the other side
  router_advertise_config = {
    groups    = []
    # Only advertise the route to the Filestore IP range
    ip_ranges = { "10.99.17.12/29" = "Filestore peering route" }
    mode      = "CUSTOM"
  }
  tunnels = {
    remote-0 = {
      bgp_peer = {
        address = "169.254.1.2"
        asn     = 64514
      }
      bgp_session_range     = "169.254.1.1/30"
      ike_version           = 2
      vpn_gateway_interface = 0
      shared_secret         = module.vpn_ha-1.random_secret
    }
    remote-1 = {
      bgp_peer = {
        address = "169.254.2.2"
        asn     = 64514
      }
      bgp_session_range     = "169.254.2.1/30"
      ike_version           = 2
      vpn_gateway_interface = 1
      shared_secret         = module.vpn_ha-1.random_secret
    }
  }
}

Et voilà, we can mount the Filestore share from a VM in project2:

➜ sudo mount -o rw,intr 10.99.17.14:/main_share /mnt/fs

➜ sudo showmount -e 10.99.17.14
Export list for 10.99.17.14:
/main_share           192.168.0.0/16,172.16.0.0/12,10.0.0.0/8
/config/google-prober 10.99.17.14

Here’s how the network request flow will go when a compute workload in project2 with private IP 10.1.0.17 requests to mount Filestore with IP 10.99.17.14 :

1. find the route to10.99.17.12/29in vpn-project2-to-project1 — next hop is the VPN GW in project2
2. go through the VPN tunnel from project2 to project1
3. find the route filestore-peer-123abc — next hop is the Filestore peering
4. on its way back, find the route back to 10.1.0.0/18 in vpn-project1-to-project2 — next hop is VPN GW on this side (project1)
5. return to the originating subnet via VPN

Hope this helps 💫 And since this is published just before Christmas 🎄— Happy Holidays! Here’s a photo of me wearing a vintage “ugly jumper” gilet and a Ukrainian embroidered shirt. A surprisingly good — and Christmassy — combo 💙💛