Planet scale applications with Kubernetes Cluster Federation

I recently developed and deployed an application in Google Cloud Platform that can scale massively while providing low latency across the globe. Here is the architecture:

A demo of the application and GCP components:

The application provided rapid autocomplete suggestions of millions of products. The data store is Redis with sorted sets of the product names. A python program that exposes an HTTP API talks with Redis. With the data store decoupled from the web application, each of these can be scaled separately. The code itself is simple but the majority of the time was spent on setting up Kubernetes Cluster Federation in GCP. Some of the GCP technologies used were:

Google Domains — Domain registration

Google Cloud DNS — DNS

Kubernetes Engine — Managed Kubernetes Clusters across the world

Cloud Storage — Static file storage

CDN — Caching of the HTTP requests

Python — Web application code (with uWSGI)

Redis — Fast in-memory data store

LetsEncrypt — Free SSL certificates for the domain

The following commands will let you to create the cluster federation easily. In the future Google Cloud might make it easier but for now, if you don’t want to spend hours working around some bugs then use this!

# Read the commands and modify according to your needs.
# Execute them in the Google cloud shell
# Set the project and DNS Zone name. Register this domain and update the NS servers after running the gcloud dns command below.

export PROJECT=<project id>
export DNS_ZONE=autocomplete.xyz.


# Getting the SSL certificates. You can also use Kube Lego for this

# sudo certbot -d autocomplete.xyz --manual --preferred-challenges dns certonly
# sudo su && cd /etc/letsencrypt/archive/
# tar cvz autocomplete.xyz.tar.gz autocomplete.xyz/

# Download them and upload it in the HTTPS load balancer

# Create the clusters
gcloud container clusters create "east-cluster" \
--zone "us-east4-b" \
--machine-type "n1-standard-1" \
--scopes "https://www.googleapis.com/auth/cloud-platform" \
--num-nodes "3" \
--enable-autoscaling --min-nodes "1" --max-nodes "4" &


gcloud container clusters create "europe-cluster" \
--zone "europe-west1-c" \
--machine-type "n1-standard-1" \
--scopes "https://www.googleapis.com/auth/cloud-platform" \
--num-nodes "3" \
--enable-autoscaling --min-nodes "1" --max-nodes "4" &

gcloud container clusters create "asia-cluster" \
--zone "asia-south1-b" \
--machine-type "n1-standard-1" \
--scopes "https://www.googleapis.com/auth/cloud-platform" \
--num-nodes "3" \
--enable-autoscaling --min-nodes "1" --max-nodes "4" &

# Download proper client bins
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.8/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz
sudo chmod +x kubernetes/client/bin/kubectl && sudo mv kubernetes/client/bin/kubectl /usr/local/bin
sudo chmod +x kubernetes/client/bin/kubefed && sudo mv kubernetes/client/bin/kubefed /usr/local/bin
sudo cp /google/google-cloud-sdk/bin/kubectl /google/google-cloud-sdk/bin/kubectl.1.8.6
sudo cp /usr/local/bin/kubectl /google/google-cloud-sdk/bin/kubectl

# Get the credentials
gcloud container clusters get-credentials east-cluster --zone=us-east4-b
gcloud container clusters get-credentials europe-cluster --zone=europe-west1-c
gcloud container clusters get-credentials asia-cluster --zone=asia-south1-b

# Rename the context for ease of use and to follow RFC
kubectl config set-context east --cluster=gke_${PROJECT}_us-east4-b_east-cluster --user=gke_${PROJECT}_us-east4-b_east-cluster
kubectl config set-context europe --cluster=gke_${PROJECT}_europe-west1-c_europe-cluster --user=gke_${PROJECT}_europe-west1-c_europe-cluster
kubectl config set-context asia --cluster=gke_${PROJECT}_asia-south1-b_asia-cluster --user=gke_${PROJECT}_asia-south1-b_asia-cluster


# Workaround for RBAC error
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account) --context east
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account) --context europe
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account) --context asia

# Initiate federation
kubefed init kfed \
--host-cluster-context=east \
--dns-zone-name=${DNS_ZONE} \
--dns-provider=google-clouddns

# Create a global ingress ip to be used by the ingress
gcloud compute addresses create ingress-ip --global

# Join the clusters to federation
kubefed --context=kfed join east-cluster \
--cluster-context=east \
--host-cluster-context=east

kubefed --context=kfed join europe-cluster \
--cluster-context=europe \
--host-cluster-context=east

kubefed --context=kfed join asia-cluster \
--cluster-context=asia \
--host-cluster-context=east

# Create namespace and verify
kubectl --context=kfed create ns default
kubectl --context=kfed get clusters

# Create the services and deployments
kubectl --context=kfed create -f master.yaml
kubectl --context=kfed create -f ingress.yaml

# Scale
# kubectl scale --replicas=9 --context=kfed deployment/redis-master
# kubectl scale --replicas=9 --context=kfed deployment/frontend

# Check Cache
# curl -s -D - -o /dev/null https://autocomplete.xyz/api?term=Go

# Sample response to test CDN
# HTTP/2 200
# server: nginx/1.13.8
# date: Thu, 12 Apr 2018 23:33:12 GMT
# content-type: application/json
# via: 1.1 google
# cache-control: max-age=86400,public
# content-length: 344
# age: 81
# alt-svc: clear

# Load testing
# hey -n 20000 -c 100 -h2 https://autocomplete.xyz/api?term=Z

# Randomise and test
# while true;do curl "https://autocomplete.xyz/api?term=$(LC_CTYPE=C tr -dc A-Za-z < /dev/urandom | head -c 2 | xargs)";done

# from file

# while true;do curl "https://autocomplete.xyz/api?term=$(sort --random-sort 2chars.txt | head -n 1)";done