Cluster Federation and Global Load Balancing on Kubernetes and Google Cloud — Part 1
WARNING: Kubernetes Federation as described in this post is no longer supported. New tools are being developed to replace it, and I’ll like to an new blog post when those tools are ready. The below should still work in theory, but I no longer recommend it.
I recently gave a talk at Google Cloud Next about using Kubernetes to deploy and manage microservices around the world using Kubernetes.
I’ve blogged about how you can set up MongoDB in a StatefulSet already, so now I’m going to deep dive on how you can set up cluster federation and deploy your services around the world!
I’m going to assume you know the basics of creating a project in Google Cloud and have installed and set up the Google Cloud SDK. If not, please check out my previous blog posts.
We are going to be looking at three things specifically, Cluster Federation, Federated Ingress, and cross cluster service discovery. I’m going to split these topics into three blog posts so look out for the next ones soon!
Part 1: Cluster Federation
Cluster Federation is what you can use to manage multiple Kubernetes clusters as if they were one. This means you can make clusters in multiple datacenters (and multiple clouds), and use federation to control them all at once!
Federated Ingress is super sweet, and is currently only supported on Google Cloud. This allows you to spin up a single load balancer with a single global IP address that will dynamically send traffic to the nearest cluster automatically!
Part 3: Cross Cluster Communication
Cross cluster service discovery is a concept that let’s services in one Kubernetes cluster find services in other clusters automatically. We will also look at Google Cloud Pub/Sub to enable asynchronous microservices!
Part 1: Cluster Federation
Step 1: Set up DNS
DNS is core to Kubernetes, as it is used for service discovery in the cluster. While running a local DNS server works inside a cluster, you need to have a central DNS service for cross cluster service discovery. You can use any programmable DNS service to do this, and we are going to use Google Cloud DNS.
The first step is to purchase a domain name (or use one you already have). In my case, I used Google Domains to buy a cheap domain name. For this example, I’m going to pretend the domain I bought is “mycompany.com”
Now, you need to create a managed DNS zone. Let’s use the “infra” subdomain to handle the Kubernetes services, so this means the domain will be “infra.mycompany.com.” I’m also going to call this zone “kfed”, but you can call it whatever you want.
gcloud dns managed-zones create kfed \
--description “Kubernetes Federation Zone” \
If you go to the Cloud DNS console, you should see this new zone created.
Click the Zone Name, and you will see more details, including the name servers:
At this point, go to your domain name registrar and use the DNS servers from the new managed zone. Here is how it looks in Google Domains:
Step 2: Create Clusters
The fun and easy step. Let’s make two small Kubernetes clusters to federate.
Create the first cluster (let’s use the us-west1 datacenter):
gcloud container clusters create west-cluster \
--zone us-west1-a --scopes cloud-platform
And now the second (let’s use the us-east1 datacenter):
gcloud container clusters create east-cluster \
--zone us-east1-b --scopes cloud-platform
After a few minutes, both clusters should be up and running.
Step 3: Create Federation Control Plane
The federation control plane is what manages the global state of all your federated clusters. The neat thing is the control plane can be self hosted inside one of your Kubernetes clusters, so you don’t even need to set up anything extra!
Note: Even if the Control Plane cluster goes down, other the clusters themselves are independent, so they will continue to function until the control plane comes back up. You can still control the individual clusters individually! This means you don’t have to worry about a single failure taking down all your clusters!
The first the you need to find is the context names for your Kubernetes clusters. Run this command:
kubectl config get-contexts
You should see output like this:
You can see that each cluster has it’s own context, and the little star indicates which context is currently active.
There is one extra step that you need to do here. Kubernetes Federation uses the context name to set up the federation secret, but it needs to conform to the RFC1123 spec. This means you need to rename the context. You can do this with the following commands:
#East Cluster kubectl config set-context east-coast \
--cluster gke_$(PROJECT_ID)_$(CLUSTER_ZONE)_east_cluster \
--user gke_$(PROJECT_ID)_$(CLUSTER_ZONE)_east_clusterkubectl config delete-context \
gke_$(PROJECT_ID)_$(CLUSTER_ZONE)_east_cluster#West Clusterkubectl config set-context west-coast \
--cluster gke_$(PROJECT_ID)_$(CLUSTER_ZONE)_west_cluster \
--user gke_$(PROJECT_ID)_$(CLUSTER_ZONE)_west_clusterkubectl config delete-context \
Replace $(CLUSTER_ZONE) and $(PROJECT_ID) as needed.
To make the process of deploying federation easier, we will use the kubefed tool. Let’s use the west-coast cluster to host the control plane (because West Coast = Best Coast).
kubefed init kfed \
Note the trailing period at the end of the dns-zone-name. This is very important!
At this point, the federation control plane will deploy. Give it a minute or two to fully deploy. This command will create a “virtual” context called “kfed.” When you use the “kfed” context, you are addressing all the federated clusters.
Step 4: Federate Clusters
The next step is to add both clusters to the federated control plane. You can use the kubefed tool to do this as well.
kubefed --context=kfed join west-cluster \
--host-cluster-context=west-coastkubefed --context=kfed join east-cluster \
With these two commands, both clusters are now federated! You can check the state of all federated clusters with this command:
kubectl --context=kfed get clustersNAME STATUS AGE
east-cluster Ready 15s
west-cluster Ready 42s
There is currently an issue with kubefed that prevents it from creating the default namespace. Run this command to fix it:
kubectl --context=kfed create ns default
Step 5: Set up Cluster DNS
Now the federation is set up, you have to make sure the DNS servers inside each cluster know to use the DNS Zone you created to perform cross-cluster service discovery.
The first step is to create a Config Map that the Kubernetes DNS controller will automatically use to connect to the DNS Zone
Replace the domain name with yours, and save the file as kubedns-config.yaml
Now create the ConfigMap:
kubectl --context=kfed apply -f kubedns-config.yaml
One important thing to remember is to use the right context when running your kubectl commands. If you run commands on the federated context, the commands usually propagate to the underlying clusters. If you run commands on an individual cluster, the commands stay within that cluster.
That’s it! You have successfully created a set of federated kubernetes clusters!
Read Part 2 where we set up Federated Ingress!