Global Kubernetes in 3 Steps on GCP

Creating a globally federated Kubernetes cluster may sound daunting but it really only takes a few small steps.

• Create the project and clusters
• Install and Join to kubefed
• Deploy globally

The kubefed utility takes most of the effort out of this process. If you want to dig in and understand what’s going on under the covers check out Kelsey Hightower’s Kubernetes Cluster Federation the hard way. Much of my write up here is patterned off Kelsey and Sandeep’s original write up just simplified and tweaked to suit my needs.

TL;DR
See the bottom for a single script to do the whole install automatically

Prerequisites

Kubefed

• This tutorial uses a utility called Kubefed to automate the federation. Get Kubefed here

DNS

• kube-dns is installed during the process and is used to setup and manage dns entries for services. To use this effectively you’ll need to have a dns zone available for use here. Make note of the zone name (with the trailing dot) and set it in the variable below

First lets set some variables for use later in the commands

export PROJECT=your-projectname
export DNS_ZONE=your.domain.com.

Step 1 — Create clusters

Create clusters in east and west

gcloud container clusters create west-cluster \
--zone us-west1-a --scopes “cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite"
gcloud container clusters create east-cluster \
--zone us-east1-b --scopes “cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite"

Get Credentials

# Workaround for RBAC error
# https://github.com/kubernetes/kubernetes/issues/42559
gcloud config set container/use_client_certificate True
export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True
# Get credentials
gcloud container clusters get-credentials west-cluster --zone=us-west1-a
gcloud container clusters get-credentials east-cluster --zone=us-east1-b

Create aliases
We need shorter names that are compliant with a specific regex that happens to exclude underscores

kubectl config set-context east --cluster=gke_${PROJECT}_us-east1-b_east-cluster --user=gke_${PROJECT}_us-east1-b_east-cluster
kubectl config set-context west --cluster=gke_${PROJECT}_us-west1-a_west-cluster --user=gke_${PROJECT}_us-west1-a_west-cluster

Step 2 — Install and Join to kubefed

Here were using kubefed to initialize the federation using the “east” context

kubefed init kfed \
 --host-cluster-context=east \
 --dns-zone-name=${DNS_ZONE} \
 --dns-provider=google-clouddns
kubectl config use-context kfed

The `kfed` context should be used for all federated deploys. You can inspect the various cluster independently but all actions should be performed on the federated context.

Join the clusters into the federation
Provide the contexts to the federation server to join in the clusters

kubefed --context=kfed join east-cluster \
--cluster-context=east \
--host-cluster-context=east
kubefed --context=kfed join west-cluster \
--cluster-context=west \
--host-cluster-context=east

Note: When adding a cluster from Europe in the federation I was unable to use the alias for some reason. I substituted the original context name back in for the -cluster-context parameter and it joined perfectly

Create the default namespace

kubectl --context=kfed create ns default

Review the clusters are joined

kubectl --context=kfed get clusters

Step 3 — Deploy your application

Lets deploy a simple application to see this in action

kubectl --context=kfed create deployment nginx --image=nginx && \
 kubectl --context=kfed scale deployment nginx --replicas=4

See it deployed automatically across both clusters

Lets view the pods for east and then for west clusters

kubectl --context=east get pods
NAME READY STATUS RESTARTS AGE
nginx-167110513–83gdg 1/1 Running 0 32s
nginx-167110513-s30vs 1/1 Running 0 32s
kubectl --context=west get pods
NAME READY STATUS RESTARTS AGE
nginx-167110513–1wrjz 1/1 Running 0 38s
nginx-167110513-v71h2 1/1 Running 0 38s

Conclusion

Nice and easy! Setting up a globally distributed app does’t need a lot of effort. With Kubenetes cluster federation you can treat multiple clusters as a single entity. In just a few simple steps the setup is complete and ready to use.

And if that wasn’t easy enough here’s all that in one script

## Requires these variables to be set
# export PROJECT=your-projectname
# export DNS_ZONE=your.domain.com.
[[ -z “$PROJECT” ]] && { echo “Env var PROJECT is missing, please export PROJECT=your-project” ; exit 1; }
[[ -z “$DNS_ZONE” ]] && { echo “Env var DNS_ZONE is missing, please export DNS_ZONE=your.domain.com.” ; exit 1; }
# Create Clusters
gcloud container clusters create west-cluster --zone us-west1-a --scopes “cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite"
gcloud container clusters create east-cluster --zone us-east1-b --scopes “cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite"
gcloud config set container/use_client_certificate True
export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True
# Get credentials
gcloud container clusters get-credentials west-cluster --zone=us-west1-a
gcloud container clusters get-credentials east-cluster --zone=us-east1-b
# Make Aliases
kubectl config set-context east --cluster=gke_${PROJECT}_us-east1-b_east-cluster --user=gke_${PROJECT}_us-east1-b_east-cluster
kubectl config set-context west --cluster=gke_${PROJECT}_us-west1-a_west-cluster --user=gke_${PROJECT}_us-west1-a_west-cluster
# Install and join the federation
kubefed init kfed --host-cluster-context=east --dns-zone-name=${DNS_ZONE} --dns-provider=google-clouddns
kubefed --context=kfed join east-cluster --cluster-context=east --host-cluster-context=east
kubefed --context=kfed join west-cluster --cluster-context=west --host-cluster-context=east
kubectl --context=kfed create ns default
# Use kfed context to create resources and deployments
kubectl config use-context kfed
# Show me the money
echo “kubectl --context=kfed get clusters “
kubectl --context=kfed get clusters