Managing Clusters with Cluster Registry

Christopher Grant
Feb 6, 2019 · 4 min read

Managing Clusters w/ Cluster Registry

The Cluster Registry is part of the kubernetes foundation but is not included in the distribution by default. It’s added through a simple kubectl apply command. The cluster registry provides a simple yet powerful feature in that it enables a new resource type of Cluster to be added to your api calls. This means you can make a request for kubectl get clusters to retrieve a list of cluster matching specific queries. Since these are presented as standard Kubernetes API requests and resources, you can combine them with other commands to more dynamically manage your ecosystem.

In this example we’ll install the registry, load some metadata then use the registry to act on multiple clusters.

While you can accomplish this on your laptop, I’ll be demonstrating through Google Cloud Shell, which already includes many of the tools we’ll be using, as well as a standard environment we can all work from.

If you’re new to GCP, Google offers a free $300 credit as well as a generous Free Tier to get you started. So create your project and lets get started

Set Variables

To get started let’s create some variables and grab kubectx to manage our contexts easier

export PROJECT=$(gcloud config get-value project)
gcloud config set project $PROJECT
export PROJECT_DIR=$HOME/clsRegistry
mkdir -p $PROJECT_DIR/bin
PATH=$PROJECT_DIR/bin/:$PATH

Get tooling

Kubectx is a fantastic tool that simplifies managing and switching cluster contexts.

curl -LO https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx && \
chmod +x kubectx && \
mv kubectx $PROJECT_DIR/bin/.

Create your control cluster

For this example we’ll use a centralized control cluster to act as our administrative system. Since we’re just doing a small example we’ll also set the number of nodes to 1

gcloud container clusters create central --zone us-central1-a --num-nodes=1
kubectx central=gke_${PROJECT}_us-central1-a_central

Install the cluster registry CRD

Once the control cluster is up and running, it’s time to create the Cluster Registry resource definition.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/cluster-registry/master/cluster-registry-crd.yaml

Generate Yaml Files describing your clusters

In general it’s good practice to store your desired state in a source repository. We’ll create yaml files describing our clusters. While we’re not doing it here, you could actually use these definitions as the specs for creating the clusters themselves.

Create 3 files, one for each cluster. In these examples I’ve included labels for lifecycle as well as the zone in a custom data structure. You can add any data needed for your use case.

cat <<EOF >>$PROJECT_DIR/central.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
name: central
namespace: default
labels:
lifecycle: "control"
spec:
kubernetesApiEndpoints:
serverEndpoints:
- clientCIDR: "0.0.0.0/0"
serverAddress: "100.0.0.0"
myCustomData:
zone: "us-central1-a"
status: {}
EOF
cat <<EOF >>$PROJECT_DIR/east.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
name: east
namespace: default
labels:
lifecycle: "prod"
spec:
kubernetesApiEndpoints:
serverEndpoints:
- clientCIDR: "0.0.0.0/0"
serverAddress: "100.0.0.0"
myCustomData:
zone: "us-east1-b"
status: {}
EOF
cat <<EOF >>$PROJECT_DIR/west.yaml
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
name: west
namespace: default
labels:
lifecycle: "prod"
spec:
kubernetesApiEndpoints:
serverEndpoints:
- clientCIDR: "0.0.0.0/0"
serverAddress: "100.0.0.0"
myCustomData:
zone: "us-west1-b"
status: {}
EOF
kubectl apply -f $PROJECT_DIR/central.yaml --context central

Create your remote cluster

OK lets spin up our clusters and load the registry. I’m also renaming the default context name here for easy use later on.

gcloud container clusters create west --zone us-west1-b --num-nodes=1
kubectx west=gke_${PROJECT}_us-west1-b_west
kubectl apply -f $PROJECT_DIR/west.yaml --context central
gcloud container clusters create east --zone us-east1-b --num-nodes=1
kubectx east=gke_${PROJECT}_us-east1-b_east
kubectl apply -f $PROJECT_DIR/east.yaml --context central

Deploy to all Prod clusters

In this section we’ll use the data in the registry along with a simple for loop to deploy an nginx image to all production clusters. We’re going to loop over the result of kubectl get clusters -l lifecycle=prodwhich selects resources labeled lifecycle=prod. Additionally we’re requesting the output to be a specific value in the json path located at .items[*].metadata.name

kubectx centralfor cluster in $( kubectl get clusters -l lifecycle=prod -o jsonpath='{.items[*].metadata.name}' ); do    kubectl run --image nginx myapp --context ${cluster}done

List all deployments on all clusters

Now let’s run a simpler command to list all the deployments from all the clusters. You’ll note myapp deployed to east and west with nothing on central

for cluster in $( kubectl get clusters -o jsonpath='{.items[*].metadata.name}' ); do
kubectx $cluster
kubectl get deploy
echo
done

Recap

So that about wraps it up. There’s really not much to setting up cluster registry but it provides a solid base for dynamically managing your fleet of kubernetes clusters.

Cleanup

Be sure to tear down your cluster to avoid additional costs

delete_cluster() {

CLUSTER_NAME=$1
CLUSTER_ZONE=$2
# Delete the clusters
gcloud container clusters delete ${CLUSTER_NAME} \
--zone=${CLUSTER_ZONE} \
-q --async
# Delete
kubectx -d ${CLUSTER_NAME}
}
delete_cluster central us-central1-a
delete_cluster west us-west1-b
delete_cluster east us-east1-b

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade