One of my clients is working on retiring some older systems and replacing them with more robust, cloud-native services. However, they still need to facilitate some interoperability between the old and the new. So, we have a handful of VMs inside GCP communicating with on-prem resources in a setup that works something like this:
The challenge I recently encountered was attempting to write some serverless resources that would also communicate across the VPN to the on-prem resources. These are simple jobs run periodically via Cloud Scheduler. The obvious choice seemed to be Cloud Run, which lets you run an arbitrary Docker container. Cloud Scheduler and Cloud Run work quite well together. Specifically, I need only my Cloud Scheduler jobs to have access to run these jobs. The challenge is that Cloud Run instances live in a strange network la-la land that isn’t connected to your GCP project’s virtual private cloud (VPC). Since the VPN gateway lives inside the VPC, this means that you can’t connect to the VPN gateway, which also means no talking to on-prem resources from Cloud Run instances. Cloud Run is off the table for me.
Luckily, Google App Engine has a “flexible environment” that will also let you run Docker containers. However, there are two problems to solve. First, App Engine apps are public by default. The first choice is App Engine firewall, which lets you configure IP address-based rules. However, this doesn’t provide tight enough security, since Cloud Scheduler’s source IP addresses originate outside the VPC as well. Instead, I needed to use Identity-aware Proxy (IAP), which lets you use Google’s Identity and Access Management (IAM) to control which specific users are allowed to access each App Engine service. Great! I set up IAP, set up a policy so that only my service account can access the app, and then… everything should work great! Except…
My Cloud Scheduler jobs continually got HTTP response codes of 401 (Unauthorized). Cloud Scheduler supports using an OIDC token and audience to authenticate its HTTP-based jobs:
I had left the Audience field blank initially, so, naturally, I went looking for the proper data to put into the audience field. Surely that was the reason it was failing, right? IAP gave me this bit of information:
“aud” is the “Audience” claim, so surely that’s correct? Actually… no. Still getting 401s. Turns out, the proper audience is actually the Client ID of the Identity Aware Proxy:
Setting up the Audience field in Cloud Scheduler with the Client ID got my Cloud Scheduler jobs working correctly.
Summing Up
If you want your serverless infrastructure in Google Cloud Platform to connect over a VPN without exposing those serverless endpoints to the entire Internet, your best option is App Engine + Identity-aware Proxy (IAP). Be sure to use the Client ID from IAP as your Audience, or IAP will reject your requests as unauthorized. Google may change how this works in the future (I sure hope so!) but for now, Cloud Run isn’t quite robust enough to handle this. Hope this helps someone!
