Cloud Run and Load balancing: go beyond your own project!

Routing and load balancing are the pillars of the Internet and its scalability. On Google Cloud, these aspects are great due to a global networks and anycast IP deployed on Global HTTPS load balancers.
Like the other cloud services, serverless compute products (App Engine, Cloud Functions and Cloud Run) have been getting the load balancing capacity few month ago.

One of the most interesting load balancing feature is the capacity to route the traffic from the Load Balancer to the deployed services the closest to the user location. And thus to have the best latency, wherever the users are.

Serverless NEG for serverless load balancing

For this, and as described in the documentation, blog and articles, you have to use the serverless NEG (Network Endpoint Group) and to add the services, deployed in different region, to it.

Use the serverless NEG as load balancer backend and enjoy!

This situation is perfect when all the services and the load balancer are deployed in the same project. And the serverless NEGs are only compliant with this case

Load balancers using serverless NEG backends must be created in the same project as the Cloud Run (fully managed), App Engine, or Cloud Functions services pointed to by the NEG.

However, the reality is often different. Sometimes, you need to reuse services from other projects for different reason:

  • Billing constraint: each project need to have its own billing because labels aren’t enough in some use cases.
  • Team organisation and deployment velocity: typically a data scientist team that serves a model usable by one or several web teams
  • Security isolation: Least privilege principle with IAM policies

And it’s especially true with Cloud Run because more and more services that we developed are hosted on it.

How to use a load balancer with Cloud Run services deployed in other projects?

Internet NEG solution

Recently, a new type of NEG has been introduced: Internet NEG. Its purpose is to route the requests from the global HTTPs Load Balancer to an internet endpoint, defined by its IP or its fully qualified domain name.

So, that can fit my use case! Instead of calling internally a Cloud Run service with a serverless NEG, we can use an Internet NEG to call it externally!

Let’s go!

Create an hello world Cloud run

  • In the menu, go to Cloud Run
  • Name it, choose your region. In the step 2 use the example container
  • Finish by allowing unauthenticated connection, for easier tests

So, now, get the service URL and test it if you want. Keep this URL, we will use it later.

Create an Internet NEG

  • In the menu, go to Compute Engine -> Network endpoint groups
  • Name it, and choose an Network Endpoint Group (Internet) type
  • Then, set the port 443 by default and select Fully qualified domain name in the Add through section.
  • Fill in with your Cloud Run service URL (without the https://) and click on create

Perfect, now the latest, and the biggest piece, let’s configure an HTTPs Load Balancer.

Create the global HTTPS Load Balancer

  • In the menu, click on Network services
  • Click on Create a load balancer
  • Click on Start configuration for the HTTPS Load Balancing
  • Expose from Internet, click on continue in the following screen
  • In the Backend configuration, select a Backend service and Create a backend service
  • On the new screen, name your backend and, as type, select Internet network endpoint group
  • Choose the protocol HTTPS to call your Cloud Run service and the name of your previously created Internet network endpoint group
  • Click on Create on the bottom

That’s over for the backend! Let’s continue the configuration
Let the Host and path rule as-is and go to the frontend configuration

  • Name it and select your protocol. You can choose HTTPS, but HTTP also work and the configuration easier (no certificate and DNS to manage).
  • Use the simplest configuration for you and click on create.

Now the load balancer is creating; wait few minutes to finish and to dispatch the configuration all around the globe.

Time to test! Go back to your Load Balancer and copy the IPv4.

Paste it in your browser and.. FAIL!!! Yes FAIL, this solution doesn’t work!
You should have a 404, and that won’t change.

The Cloud Run routing

The issue comes from the Cloud Run routing mode. The URL provided by Cloud Run service isn’t bind to an IP or a CNAME, the routing is based on the host value of the request’s header.

It could be surprising, let’s validate this and have a try with curl directly on the root domain of Cloud Run: And we will add only the Cloud Run service fully qualified name (URL without https://) in the host header

curl -H "host: <Cloud Run fully qualified name>"

It works!! And yes, you can reach any Cloud Run services with this solution, you only have to know the exact URL name of the service.
In fact, it’s not really surprising because Kubernetes and Knative works on the same way.

Now that we know the routing mode, we can go back to the previous configuration and have a fresh look on the issue:

The host in the request’s header is the load balancer IP address (or your own domain name if you set up it).

Therefore, to solve this, we have to override the host header in the backend configuration of our Load Balancer with backend custom header.

Backend custom header

  • Go back to the load balancer, edit it and go to your backend.
  • On the bottom, click on Advanced configuration
  • Then click on Add header
  • And add host as key, and the Cloud Run fully qualified name of your service as value
  • And click on Update
  • And again on Update in your load balancer edit page

Here again, wait few minutes for the configuration advertising. And test again your IP.

Boom! Now that works!!

That’s great for 1 service, but how to do with several Cloud Run services?

Multi Cloud Run service routing

When you have several Cloud Run services to call with a load balancer like this, in the same other project or in different project, the immediate idea is to repeat the same pattern for each service

  • Create an Internet NEG
  • Create a backend
  • Customize the host and path rules

Actually, you don’t need to create an Internet NEG per service, you can cheat!

Indeed, because all the Cloud Run services requests land on the URL, you can create only one Internet NEG, and use it in multiple backend definition where you customize only host request’s header to route the traffic to the correct Cloud Run service.

Let’s deploy this!

Create a new service

Start by creating a new Cloud Run service, as previously done. You can use the same example container image.

Get the URL, we will need it later

Update your “generic” Internet NEG

  • Go back your internet NEG
  • You can’t update the current endpoint. So, select it and click on Remove endpoint; Confirm the deletion
  • And click on Add network endpoint.
  • Add simply the value:
  • And click on Create

And nothing appear. Wait few seconds, the end of the creation and your endpoint appears! It’s an UI glitch.

Add a second backend

  • Go back to your load balancer, clieck on Edit and go to the backend part.
  • Create a backend service and, as previously done, select the Internet NEG type, the HTTPS protocol, select your generic Internet NEG and go to Advanced option to override the host header with the Cloud Run service fully qualified name of your 2nd service.
  • And hit Create

Customize the host and path rules

This customization can be complex here. But here my requirement:

  • I want to reach the 2nd Cloud Run service on the path /hello2
  • The second Cloud Run service doesn’t have a path /hello2. So I need to rewrite the path to /

Let’s configure this!

  • Select Advanced host and path rule radio button and click on Add host and path rule
  • Firstly, define the host name filtering. In my case, don’t care. I set *
  • Then, click on the pencil of the default path, and select the backend by default. This one which answer on the / path on the load balancer
  • And Save
  • Then, click on Add path rule to configure the second service
  • Add the path, in my case /hello2 and select my second backend.
  • But, I need to rewrite the path. For this, click on Add-on action and, on Path prefix rewrite set /
  • And click on Save
  • And click on Update to save the new configuration of your load balancer

Wait again few minutes and test your IP on the / path and on the /hello2 path.

Congrats!!! You have a load balancer with several Cloud Run services deployed in different projects!!

Routing to the closest location

This configuration is good, but you loose one of the main advantage of the a global HTTPS load balancer: the capacity to route the request to the closest location.

In this case, you have to combine

  • The Serverless NEG: for the multi-region service as backend of a HTTPS load balancer and all in the same project
  • The Internet NEG: to access externally to the load balancer of the multi-region Cloud Run services.

Because of the constraint of the Serverless NEG, the solution works only if the multi-region Cloud Run services are deployed on the same project.
In other words, you can’t have configuration with a project per region.

To achieve this, 2 main steps:

  1. Firstly, create a HTTPS load balancer in the multi-region project. Use standard Serverless NEG. Here, it’s important to use the HTTPS protocol for the frontend configuration.
    It’s the documented configuration.
  2. Then, create another HTTPS load balancer, in another project, with an Internet NEG that use the IP or the domain name of the multi-region HTTPS load balancer.
    This time, no need to override the host request’s header value because you don’t reach directly the Cloud Run services, but only another load balancer which will perform this operation automatically through the Serverless NEG.

Because the communication between the Internet NEG and the multi-region HTTPS load balancer is over the Internet, HTTPS is strongly recommended in the documentation

We strongly recommend you use HTTPS or HTTP/2 as the protocol when configuring a backend service with an internet NEG, so that communication between the external HTTP(S) load balancer and your backend is encrypted and authenticated when transiting the public internet.

Go beyond the limits!

Even if these solutions seem easy to implement and use, it’s important to keep in mind that each hop (through a load balancer) implies:

  • Latency, even few milliseconds.
  • New possible point of failure in your architecture
  • Additional cost for the load balancers

Eventually, the project-only limitations of Serverless NEG are not limits thanks to Internet NEG capacity. And with a composition of load balancers, request headers and URL mapping, it’s possible to address all the use cases!



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
guillaume blaquiere

guillaume blaquiere


GDE cloud platform, Group Data Architect @Carrefour, speaker, writer and polyglot developer, Google Cloud platform 3x certified, serverless addict and Go fan.