Multi Region Load Balancing with GO and Google Cloud Run — Part 3
In part 3, we will be testing our new multi region load balanced Cloud Run service and ensuring that it’s doing what we expect.
Introduction
So in part 2, we provisioned all the infrastructure that we required via terraform. We now have a service running on Cloud Run in four regions across the world behind a HTTPS load balancer. In this final instalment we will be carrying out some checks and tests on our new service to make sure that the load balancing is behaving as we expect. We are going to test 2 things in this tutorial.
- That the cloud run instance closest to our location services our requests.
- If our closest region goes down, we are served by our next nearest region
Let’s get started
So we are going to test our load balancing by two methods. First we will hit our service from our current location via a browser or Postman and check that the region returned is the closest based on the four that are running across the world.
I’m based in the UK and so I expect my closest region to be europe-west1 which is in Belgium. Currently Cloud Run is not available in all regions and so at the time of writing, Belgium is the nearest region for me with Cloud Run functionality.
As you can see from the Postman screenshot above, I am correctly served by my nearest region in Belgium!
However, this could be fluke or maybe our load balancing is only working correctly for this region. We should probably check a couple of the other regions to ensure this is actually doing what we think it is.
“Not all those who wander are lost”
Let’s take a very quick digital trip to Hong Kong and Las Vegas! We are going to create two virtual machines in these regions. This will allow us to test our load balancing from other exotic places across the world without needing to pack a toothbrush or put up with the rubbish food served on aeroplanes. :)
A trip to Hong Kong
Login in the Google Cloud Console and head to Compute Engine -> VM Instances control panel. Once there, click Create and setup and new instance with the below settings:
Name: test-hong-kong
Region: asia-east2
Machine series: N1
Machine type: f1-micro
SSH keys: Add your own public SSH keyAll other settings can be left at their default settings. If you’d like to keep the instances up and running after we are done for some reason, then I suggest that you also set your instance’s Preemptibility to on. This will cost you less than a standard instance but will be recycled randomly once every 24 hours. Since we are using these tiny instances to test on, we don’t really care if they go down.
And here is the Preemptibility setting should you want it.
Fear and Loathing in Las Vegas
Great! Now let’s repeat the exact same steps as above to create an instance in Las Vegas!
Name: test-vegas-baby
Region: us-west4
Machine series: N1
Machine type: f1-micro
SSH keys: Add your own public SSH key
Why Vegas and Hong Kong?
The eagle eyed among you might have noticed that we actually picked two regions where we don’t actually have any Cloud Run instances running! We are doing this because we want to ensure that regardless of our global location, the instance nearest to us is still picked to serve our request. We also don’t want to potentially hijack or skew our own tests by making a request from a VM located in the same region as the running service. Google may have all kinds of fancy networking or system configurations within an individual region to improve performance or something that may inadvertently affect what we are trying to test.
Extra homework
For those who do want to create a third VM in a region where we have an instance running, please do so now. You shouldn’t take my word for it that it works! :)
Let’s test our Chinese
First, we are going to test our instance running in Hong Kong. The nearest running instance to us here would be Tokyo in asia-northeast1 .
So let’s grab our new VM’s external IP address and SSH into it!
ssh -i ~/.ssh/id_rsa me@xxx.xxx.xxx.xxxOnce you are logged into the instance, we’ll want to install jq . This is a useful tool for formatting, filtering and searching through JSON data. Since our API returns JSON data we might as well use jq to format the response into something a little more readable.
sudo apt-get install jqOnce jq installed, let’s hit our running service and check the response!
curl https://your-domain.com | jqIf all is well you should see that the request was served from Tokyo!
Married by an Elvis
Next, we are going to test our instance running in Las Vegas. The nearest running instance to us here would be Oregon in us-west1 .
So let’s grab our new VM’s external IP address and SSH into it!
ssh -i ~/.ssh/id_rsa me@xxx.xxx.xxx.xxxAnd install jq again…
sudo apt-get install jqFinally, let’s hit our service from Vegas
curl https://your-domain.com | jqWe should see that our request was served from Oregon! Awesome-sauce.
Houston, we have a problem
Next up we want to test that if the cloud run instance in our nearest region is unavailable for some reason, that Google will be smart and route to the next nearest region.
Now, this is very tricky to simulate and Google doesn’t seem to provide any tools for simulating the outage of an entire region in Cloud Run. Don’t forget, this would have to be an entire region outage as Cloud Run services are automatically replicated across multiple zones within the selected region!
So for our test, we are going to do something a lot dumber. This is absolutely NOT an accurate test of the real life scenario, but its the best we can do and should allow us to see a replicated fail over which will be cool.
We are going to simulate our regional outage in Belgium (my closest region from the UK) by temporarily removing the Serverless Network Endpoint Group from our load balancer’s backend. This would simulate (to a point) a region being completely unavailable to our load balancer.
So let’s do this, run the command below (replacing relevant details with yours) to remove the serverless NEG for your nearest region.
gcloud beta compute backend-services remove-backend \
my-service-backend-service \ --global \ --quiet \ --network-endpoint-group=europe-west1-serverless-neg \ --network-endpoint-group-region=europe-west1
From your load balancer in Google Cloud Console, you should now see that there are only three endpoints in our load balancer’s backend! OH NO!
Patience, young Padawan.
Unlike a real regional outage, our outage might actually take 5–10 minutes to apply. Changes to load balancers on GCP can take a few minutes to fully take effect — most likely due to all the amazingly clever technology in the background that makes these global load balancers so blazingly fast, scalable and reliable. :)
Graceful failover
Once these changes take effect you should now see a request from your local machine go to your next nearest region — which for me here in the UK is in South Carolinaus-east1 . It works!!!
Google’s back up!
Ok, so Google have been awesome and fixed the issue with my closest cloud run region and we now have availability again! Let’s replicate this by bringing our missing serverless NEG back into the load balancer.
gcloud beta compute backend-services add-backend \ my-service-backend-service \ --global \ --network-endpoint-group=europe-west1-serverless-neg \ --network-endpoint-group-region=europe-west1
We should now have all four serverless NEGs back in our load balancer.
Again, we’ll need to wait a few minutes for these changes to apply. However, once they do you should start getting served by your nearest region again! #Winning.
We are done!
So this my friends, marks the end of our journey together! In order to not incur any additional charges on Google Cloud Platform, I recommend that you tear down all the infrastructure and then remove the project from your Google Cloud account. To do this, go back to the terraform repository from part 2 and run the command below.
terraform destroyThis will tear down all the infrastructure created by terraform leaving you with a nice empty project. Then the final step is to go into your project settings and shut it down. Google will then delete the project within 30 days if I remember correctly.
Let’s wrap it up!
Awesome. So over the last 3 articles we have:
- Written a stateless API micro service in
golang. - Packaged that service up in to a small, secure container image.
- Deployed the app to four regions across the globe.
- Setup a load balancer and serverless network endpoint groups to load balance between all the various regions.
- Secured our connection with a Google Managed SSL certificate for a custom domain.
- Tested our load balancer from our nearest location and then from Hong Kong and Vegas.
- Replicated a region outage on Google in order to see what a failover might look like!
I hope you all enjoyed the articles and learned something cool! Depending on the sort of feedback I get, I may continue writing these guides so please do share and comment!
Rob
