How to securely invoke a Cloud Function from Google Kubernetes Engine running on another GCP project

In complex environments where different teams run their own Google Cloud projects, it is challenging to make sure that a service in a project can be only accessed by specific applications running on other Google Cloud Projects. Complicated VPC peering and internal load balancing schemes are oftentimes unavoidable and sometimes it is even not possible to achieve a cross-project communication without exposing services to public Internet where multiple regions are involved.