GCP Cross-region internal application load balancer : why and how
Context :
A Google Cloud internal Application Load Balancer is a proxy-based layer 7 load balancer that enables you to run and scale your services behind a single internal IP address. The internal Application Load Balancer distributes HTTP and HTTPS traffic to backends hosted on a variety of Google Cloud platforms such as Compute Engine, Google Kubernetes Engine (GKE), and Cloud Run. This load balancer is available in 2 flavors :
- Regional internal application load balancer
- Cross-regional internal application load balancer
The cross-region internal application load balancer enables you to load balance traffic to backend services that are globally distributed, including traffic management that ensures traffic is directed to the closest backend.
The idea of this blog post is to see how cross-regional internal application load balancer is deployed and its use-cases under steady state and failover situations.
Failover may fall in 2 categories : (a) failover of a regional backend (b) failover of iLB frontend
- Failover of Frontend: When the proxy tasks in a region (where the forwarding rule is configured) fail. Failover to a forwarding rule in a different region is done using a DNS failover policy.
- Failover of Backend: When all of the backends (e.g. VM instances) in a region fail, the load balancer automatically chooses healthy, available backends in the next closest region.
Topology
In the test topology used in this blog , I had one VPC with subnets in 2 regions : asia-south1 ( Mumbai) and asia-south2 ( Delhi ) . The CIDR used in each subnet is shown in the topology diagram below . These 2 subnets host the instance groups where web application servers are hosted.
As a next step ( and a mandatory step ) for setting up cross regional internal application load balancer , it is necessary to reserve PROXY subnet for each region in context . In my example , I have reserved 2 proxy subnets ( one each for asia-south1 and asis-south2 )
Setup Load balancer : Lets break this in 3 simple parts
- Front end setup
- Backend setup
- Routing rules
Front End : Create one front end per region . You may choose the protocol as HTTP or HTTPs for the front end . In our case , we had following configured for the front end
Backend
Created one backend service
In this backend service → I created 2 backends . One for each regional instance group.In my case , each backend consist of instance groups running web server.
Sample output of load balancer backend service
$ gcloud compute backend-services describe cross-region-bserv-for-ilb --global
affinityCookieTtlSec: 0
backends:
- balancingMode: UTILIZATION
capacityScaler: 1.0
group: https://www.googleapis.com/compute/v1/projects/gm-test-337806/zones/asia-south1-c/instanceGroups/lb-test-mumbai-umig
maxRatePerInstance: 100.0
maxUtilization: 0.8
- balancingMode: UTILIZATION
capacityScaler: 1.0
group: https://www.googleapis.com/compute/v1/projects/gm-test-337806/zones/asia-south2-a/instanceGroups/lb-test-delhi-01
maxRatePerInstance: 100.0
maxUtilization: 0.8
connectionDraining:
drainingTimeoutSec: 300
creationTimestamp: '2024-04-27T21:04:05.231-07:00'
description: ''
fingerprint: WRypjLxKN0M=
healthChecks:
- https://www.googleapis.com/compute/v1/projects/gm-test-337806/global/healthChecks/auto-health
id: '1322192009993155674'
kind: compute#backendService
loadBalancingScheme: INTERNAL_MANAGED
localityLbPolicy: ROUND_ROBIN
logConfig:
enable: false
optionalMode: EXCLUDE_ALL_OPTIONAL
name: cross-region-bserv-for-ilb
port: 80
portName: lb-test-mumbai-http
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/gm-test-337806/global/backendServices/cross-region-bserv-for-ilb
sessionAffinity: NONE
timeoutSec: 30
usedBy:
- reference: https://www.googleapis.com/compute/v1/projects/gm-test-337806/global/urlMaps/cross-regional-internal-layer7TEST # 1
Client sending traffic to regional front-end :In this test , we assumed that regional clients will send traffic to respective iLB front end ; i.e Mumbai client sent traffic to ilB front end in Mumbai (asia-south1) and Delhi client sent requests to Delhi (asia-south2) front end
Test initiated from TEST-CLIENT-2 (Delhi region)
This worked as expected . Customer relied on making separate entries in DNS for each front end and this was used by respective clients in their application access method. The load balancer served traffic from closest healthy backend available.
TEST # 2
Client sending traffic to cross-regional front-end : This is continuation of test case 1 , where the requirement is to access a workload in a cross-region ; i.e. client in asia-south1 (Mumbai) trying to access workload in asia-south2 by accessing front end name / IP of layer7 iLB in asia-south2 region
Hence , once the requests land on a iLB frontend , it tries to route it to a healthy backend nearest to it
TEST # 3
Failover Scenario : In this case , let’s assume service in asia-south2 failed ( Or all VMs down ) . For simulation in my setup , I removed the backend in Delhi region from the iLB backend service configuration.
Lets see the results
Hence , as expected , in event of no healthy backend , the iLB will redirect the traffic to healthy backends in next closest region.
TEST # 4
Combining Cross region ILB functionality with GCP CLOUD DNS GEO routing POLICY
This is most interesting usecase . Here ; instead of using separate domain names for application hosted in asia-south1 / asia-south2, the customer created a domain name called “application.myapp.com”. In addition to this , customer created a GEO routing policy specifying region as ‘asia-south1’ resolve to 10.10.152.10 [ load balancer front end in asia-south1 ] and asia-south2 resolve ot 10.10.151.11 [ load balancer front end in asia-south1 ] . Sample DNS configuration is as follows -
When client attempted to access the app behind application.myapp.com ; the response came from the nearby region
BACKEND FAILOVER ( no healthy backend in one of region )
Again , as expected , no matter which frontend IP the query lands , iLB send the request to nearby healthy backend. For example , if client in Delhi region sends request to front-end in Delhi-region of iLB , this will be routed to next near healthy backend ( which is Mumbai in my setup )
TEST # 5
This test has to do with failover of proxy task in a given region . This is where Failover Policy of Cloud DNS can come handy.
In this setup , customer needs to serve everything from Asia-south1 application instance and if this is not available / reachable ; asia-south2 instance shall be considered .
Lets look at Cloud DNS Failover Policy
This simply means that clients will resolve to iLB frontend IP address in Asia-south1 ( 10.10.152.10 ) and if this isnt reachable ; clients in both regions will resolve app1.myapp.com to iLB front-end IP in asia-south2 ( 10.10.151.12 )
Hence in steady state the setup will work like follows
However in case of Front end fail , the setup will work like following
Useful URLs
Setup cross-regional internal L7 load balancer
Disclaimer: This is to inform readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.
