How to securely invoke a Cloud Function from Google Kubernetes Engine running on another GCP project
In complex environments where different teams run their own Google Cloud projects, it is challenging to make sure that a service in a project can be only accessed by specific applications running on other Google Cloud Projects. Complicated VPC peering and internal load balancing schemes are oftentimes unavoidable and sometimes it is even not possible to achieve a cross-project communication without exposing services to public Internet where multiple regions are involved.
We have recently came across an interesting challenge where a team wanted to expose their internal API as a Cloud Function and wanted to make sure that only authorized application running on neighboring teams’ GCP projects could invoke this function. They also wished a solution that requires minimum changes in their clients’s application code.
The team did not only want the ability to run this internal API in a serverless environment, their biggest goal was to avoid a complicated internal load balancing setup and to offload all the service-to-service authentication to Google. In this way, they would not have to manage complicated firewall rules, communicate with clients when these rules require changes, manage an identity solution and take the risk of managing credentials across different teams.
It is perfectly possible to invoke a Cloud Function using a service account from another GCP project. When combined with technologies like Workload Identity on GKE, it is possible to allow a Cloud Function to be invoked only by one specific application running in any GKE cluster in any project, and manage all this without managing keys yourself and requiring big client side changes.
Invoking Cloud Functions in a project where everything else also run is a very well-defined and documented problem, however there is very little information about how to orchestrate multiple projects, in a production-ready fashion.
In this tutorial we are gonna create a private Cloud Function in a GCP project. Then, we are gonna authorize a service account from another GCP project and see how we can invoke the Function using this service account. We are gonna see how to do this programatically and finally put pieces together to achieve fully managed authentication setup.
Preparation
To complete this tutorial, you need two Google Cloud projects. We are going to call one of them as Host Project and the other one as Client Project. The first one is where you are going to create a project and the other one is where your application and its service account are.
You are going to need Cloud Functions Admin (cloudfunctions.admin) role in the Host Project to be able to set the IAM policy of the function. You need Service Account Admin (iam.serviceAccountAdmin) and Service Account User (iam.serviceAccountUser) roles in order to create service accounts and their keys in the Client Project.
Creating the function
Navigate to the Host project and create a new Cloud Function on GCP console. Select HTTP as trigger. Make sure that you leave Allow unauthenticated invocations unchecked to allow invocation only with a valid GCP access token. If you cannot uncheck the box, it means that you do not have Cloud Functions Admin role assigned on your account.
In couple of seconds, you will be presented the url to access this function on the console. This URL is in the following format:
https://REGION-HOST_PROJECT_ID.cloudfunctions.net/FUNCTION_NAME
Run
curl https://REGION-HOST_PROJECT_ID.cloudfunctions.net/FUNCTION_NAME
to make sure that you are not allowed to invoke the function.
Creating a service account in another project
Use the following command to create a service account in the client project:
gcloud iam service-accounts create cloud-invoke-test \
--project CLIENT_PROJECT_ID
Create a JSON key for the service account:
gcloud iam service-accounts keys create credentials.json --iam-account \
cloud-invoke-test@CLIENT_PROJECT_ID.iam.gserviceaccount.com
Running the function with service account credentials
Activate the service account for your gcloud:
gcloud auth activate-service-account --key-file=credentials.json
Now, your gcloud command is acting as the service account. The bearer token you need to invoke the function is obtained via the following command:
gcloud auth print-identity-token
Make sure that you receive a token after running the command.
Now, you can add this token to the request and try invoking the function:
curl https://REGION-HOST_PROJECT_ID.cloudfunctions.net/FUNCTION_NAME -H
"Authorization: Bearer $(gcloud auth print-identity-token)"
This request will fail since the Cloud Function is still not configured to accept requests from this service account.
Granting permissions to the Function
Since you are Cloud Functions Admin, you are able to set IAM policies for each Cloud Function. Now you can assign roles/cloudfunctions.invoker permission to the service account from the client project via following command:
gcloud alpha functions add-iam-policy-binding FUNCTION_NAME \
--region REGION --project HOST_PROJECT_ID --member serviceAccount:cloud-invoke-test@CLIENT_PROJECT_ID.iam.gserviceaccount.com \
--role roles/cloudfunctions.invoker --account YOUR_EMAIL
Please note that we had to add the — account parameter because in the previous step, you activated the service account as your default identity. However, the service account does not have permissions to set IAM policy on the Cloud Function, so we have to act as your user account.
After couple of minutes, when permissions are settled, run again the curl command with the access token:
curl https://REGION-HOST_PROJECT_ID.cloudfunctions.net/FUNCTION_NAME -H "Authorization: Bearer $(gcloud auth print-identity-token)"
You are gonna see the output of the Cloud Function.
Programmatically invoking the Function with a token (on-premise)
For applications running on on-premise environment, it is possible to distribute the service account key along with the application and use use Google’s auth library to generate the token. Below you can find an example in Node.js:
Please note here the audience being equal to the invoke URL upon fetching an access token via service account key.
See the the Google Cloud IAP Documentation for more examples in other programming languages.
Programmatically invoking the function from GKE workloads using Workload Identity
If your application is working on GKE (or GCE with compute metadata), it is possible to fetch the access token directly from the Metadata API. In this case, you do not have to use the Google’s Auth libraries but you can obtain the access token via plain HTTP and add it to your request to the Cloud Function.
When your application runs on GKE, you can use Workload Identity to decide which Kubernetes Pod (actually Kubernetes Service Account this Pod is running with) can assume which GCP role.
Since the documentation is detailed enough, we skip here explaining how to setup the Workload Identity for a Pod. Once you complete the setup and run a Pod with the GCP service account you created before, the following URL will return the access key from inside the Pod:
http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://REGION-HOST_PROJECT_ID.cloudfunctions.net/FUNCTION_NAMEYour application can access to this URL and fetch the access token, then add this token to the Cloud Function request to authenticate itself.
Please note that this method does not use the service account key you created previously. You actually can and should delete all the service account keys you created since all the idea behind this setup is not having to manage credentials.
Please see Google’s document for detailed examples.
Conclusion
If you are a service owner exposing an API as Cloud Function, you can ask your clients to create GCP service accounts and assign them to the GKE workload they are gonna access your function from. Once you know which GCP service account e-mail this application will use, you can go ahead and add this e-mail to your function’s IAM policy. In this way, you are not gonna have to manage (even create) keys, their security and rotation. You will always make sure that your function can be only called by this service account and your client can move this account around to allow more applications to access your function. You are not gonna need to communicate to manage IP blocks, negotiate credentials and work on all the tiresome job of production readiness, getting all this time backto focus on your business instead. Moreover, no more than one HTTP call is needed to adapt the client application.
