Kubernetes Engine (GKE) multi-cluster life cycle management series

Part V: GKE multi-cluster upgrades using Ingress for Anthos step-by-step tutorial

Published in

Google Cloud - Community

18 min readApr 22, 2020

GKE Lifecycle Management using Ingress for Anthos

This tutorial describes how to upgrade a multi cluster GKE environment using the rolling upgrade strategy (discussed in the previous blog) and the multi cluster ingress feature provided by Ingress for Anthos. This tutorial is the culmination of the concepts covered in this blog series.

GCP recommends using the GKE Autoupgrade feature to upgrade your GKE clusters. Autoupgrade is a no-ops way of getting your clusters (master and nodes) automatically updated on a release schedule determined by GCP. This requires no intervention from the operator. However, in some cases, operators may want more control over how and when they upgrade clusters. This tutorial walks through a method of upgrading multiple clusters where your applications run on all clusters. It then utilizes the Ingress for Anthos (multicluster Ingress) feature to drain one cluster at a time before upgrading.

Prerequisites

This tutorial requires multiple clusters (two or more) with the same applications (namespaces, deployments, services etc) running on all clusters.
This tutorial assumes autoupgrade feature has been turned off for the clusters to be upgraded.
This tutorial uses Ingress for Anthos. Ingress for Anthos requires:

GKE static versions 1.14.10-gke.17+ or GKE clusters in the Rapid or Regular release channels.
Clusters in VPC-Native (Alias IP) mode. For more information, see Creating a VPC-native cluster.
Have HTTP load balancing enabled (enabled by default).
gcloud — version must be 281 or higher. GKE cluster registration steps depend on this version or higher.

Topology

This tutorial uses the following topology. There are a total of three clusters. Two clusters (blue and green) act as identical clusters with the same application deployed. One cluster (ingress-config) acts as the control plane cluster which configures Ingress for Anthos. In this tutorial, you deploy a sample app called the Hipster Shop app on two application clusters (blue and green clusters). Hipster Shop app is a sample microservices application comprising ten microservices simulating an e-commerce application. It consists of a web frontend (that clients can access) and a number of backend Services for example shopping cart, product catalog and recommendation services simulating a cloud-native e-retailer.

Objectives

Create three GKE clusters.
Register the clusters to GKE Hub.
Configure one GKE cluster (ingress-config) as the central config cluster. This cluster acts as the control plane for multi cluster ingress. It creates and manages the resources for Ingress for Anthos.
Deploy a sample application to the other GKE clusters. Use the Hipster Shop app as the sample application.
Configure Ingress for Anthos to send client traffic to the Hipster Shop running on both application clusters.
Set up a load generator to Hipster Shop app and configure monitoring with Cloud Monitoring.
Remove (drain) one application cluster from multi cluster ingress.
Upgrade the drained cluster.
Spill traffic back to the newly upgraded cluster using Ingress for Anthos.

Costs

This tutorial uses the following billable components of Google Cloud Platform:

GKE
Networking
Load Balancing
Cloud Monitoring

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

Before you begin

Select or create a GCP project.

GO TO THE MANAGE RESOURCES PAGE

Enable billing for your project.

ENABLE BILLING

You run all the terminal commands in this tutorial from Cloud Shell. Open Cloud Shell:

OPEN CLOUD SHELL

Set your default project.

export PROJECT=$(gcloud info --format='value(config.project)')
gcloud config set project ${PROJECT}

Enable the GKE, GKE Hub and multiclusteringress APIs in the project.

gcloud services enable container.googleapis.com gkehub.googleapis.com multiclusteringress.googleapis.com

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See Cleaning up for more detail.

Set up the environment

You run all the terminal commands in this tutorial from Cloud Shell.

Open Cloud Shell:

OPEN CLOUD SHELL

Clone the repo to get the files for this tutorial. Create a WORKDIR. This way you can delete the WORKDIR at the end of this tutorial.

cd ${HOME}
git clone https://github.com/ameer00/gke-multicluster-upgrades.gitcd gke-multicluster-upgrades
export WORKDIR=`pwd`

Create and register GKE clusters to GKE Hub

Create GKE clusters

Create three GKE clusters.

gcloud container clusters create ingress-config --zone us-west1-a --num-nodes=3 --enable-ip-alias --asyncgcloud container clusters create blue --zone us-west1-b --num-nodes=4 --enable-ip-alias --asyncgcloud container clusters create green --zone us-west1-c --num-nodes=4 --enable-ip-alias

Wait a few minutes until all clusters are successfully created. Ensure that the clusters are running:

gcloud container clusters listOutput (do not copy)NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUSingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 RUNNINGblue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 RUNNINGgreen us-west1-c 1.14.10-gke.17 35.197.46.200 n1-standard-1 1.14.10-gke.17 4 RUNNING

Master/node version and IP addresses may be different.

Create a new kubeconfig file and connect to all clusters to generate entries in the kubeconfig file:

touch gke-upgrade-kubeconfig
export KUBECONFIG=gke-upgrade-kubeconfiggcloud container clusters get-credentials ingress-config --zone us-west1-a --project ${PROJECT}gcloud container clusters get-credentials blue --zone us-west1-b --project ${PROJECT}
gcloud container clusters get-credentials green --zone us-west1-c --project ${PROJECT}

You use the kubeconfig file to create authentication to clusters by creating a user and context for each cluster. After you create the kubeconfig file, you can quickly switch context between clusters.

Verify that you have three clusters in the kubeconfig file:

kubectl config view -ojson | jq -r '.clusters[].name'Output (do not copy)
gke_gke-multicluster-upgrades_us-west1-a_ingress-configgke_gke-multicluster-upgrades_us-west1-b_bluegke_gke-multicluster-upgrades_us-west1-c_green

Get the context for the three clusters for use later.

export INGRESS_CONFIG_CLUSTER=$(kubectl config view -ojson | jq -r '.clusters[].name' | grep ingress-config)
export BLUE_CLUSTER=$(kubectl config view -ojson | jq -r '.clusters[].name' | grep blue)
export GREEN_CLUSTER=$(kubectl config view -ojson | jq -r '.clusters[].name' | grep green)
echo -e "${INGRESS_CONFIG_CLUSTER}\n${BLUE_CLUSTER}\n${GREEN_CLUSTER}"Output (do not copy)
gke_gke-multicluster-upgrades_us-west1-a_ingress-configgke_gke-multicluster-upgrades_us-west1-b_bluegke_gke-multicluster-upgrades_us-west1-c_green

Register GKE clusters to GKE Hub

GKE Hub enables you to operate your Kubernetes clusters in hybrid environments. GKE Hub also enables registered clusters to use advanced GKE features for example Ingress for Anthos. You need a GCP Service Account (SA) to register a GKE cluster to GKE Hub. Give IAM permission to the GCP SA to be able to register to GKE Hub.

Create a GCP service account and download its credentials to use for registering the cluster.

gcloud iam service-accounts create ingress-svc-acctgcloud projects add-iam-policy-binding ${PROJECT} \--member="serviceAccount:ingress-svc-acct@${PROJECT}.iam.gserviceaccount.com" \--role="roles/gkehub.connect"
gcloud iam service-accounts keys \create ${WORKDIR}/ingress-svc-acct.json \--iam-account=ingress-svc-acct@${PROJECT}.iam.gserviceaccount.com

Get the URIs from the GKE clusters.

export INGRESS_CONFIG_URI=$(gcloud container clusters list --uri | grep ingress-config)
export BLUE_URI=$(gcloud container clusters list --uri | grep blue)
export GREEN_URI=$(gcloud container clusters list --uri | grep green)
echo -e "${INGRESS_CONFIG_URI}\n${BLUE_URI}\n${GREEN_URI}"Output (do not copy)
https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-a/clusters/ingress-confighttps://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-b/clusters/bluehttps://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green

gcloud container hub memberships register ingress-config \ --project=${PROJECT} \ --gke-uri=${INGRESS_CONFIG_URI} \ --service-account-key-file=${WORKDIR}/ingress-svc-acct.jsongcloud container hub memberships register blue \ --project=${PROJECT} \ --gke-uri=${BLUE_URI} \ --service-account-key-file=${WORKDIR}/ingress-svc-acct.jsongcloud container hub memberships register green \ --project=${PROJECT} \ --gke-uri=${GREEN_URI} \ --service-account-key-file=${WORKDIR}/ingress-svc-acct.json

Verify that the clusters are registered.

gcloud container hub memberships listOutput (do not copy)
NAME EXTERNAL_IDblue d40521d9–693f-11ea-a26c-42010a8a0010green d3027ecd-693f-11ea-ad5f-42010a8a00a9ingress-config bb778338–693f-11ea-a053–42010a8a016a

Configure the ingress-config cluster as the config cluster for Ingress for Anthos. You do this by enabling multiclusteringress feature through the GKE Hub.

gcloud alpha container hub features multiclusteringress enable \ --config-membership=projects/${PROJECT}/locations/global/memberships/ingress-config

Running the command above adds the MulticlusterIngress and MulticlusterService CRDs to the config cluster.

The previous command takes a few minutes to complete. Run the following command to verify that the ingress-cluster has been successfully configured for Ingress for Anthos. Wait a few moments until you see code: OK for the ingress-config cluster (as shown below).

watch gcloud alpha container hub features multiclusteringress describeWait until the output is as shown below (do not copy)
createTime: ‘2020–03–18T18:13:46.530713607Z’featureState:details:code: OKdescription: Multicluster Ingress requires Anthos license enablement. Unlicensedusage is unrestricted for the MCI Beta API. Note that licensing will be enforcedfor use of the Generally Available MCI API.detailsByMembership:projects/960583074711/locations/global/memberships/blue:code: OKprojects/960583074711/locations/global/memberships/green:code: OKprojects/960583074711/locations/global/memberships/ingress-config:code: OKlifecycleState: ENABLEDmulticlusteringressFeatureSpec:configMembership: projects/gke-multicluster-upgrades/locations/global/memberships/ingress-configname: projects/gke-multicluster-upgrades/locations/global/features/multiclusteringress

While lifecycleState is marked as enabled, this feature is not ready to use until your config cluster has an OK status. If you do not see code: OK after a few minutes, go to the troubleshooting section. When the feature is not ready you should see the following output.

featureState:details:code: OKdescription: Multicluster Ingress requires Anthos license enablement. Unlicensedusage is unrestricted for the MCI Beta API. Note that licensing will be enforcedfor use of the Generally Available MCI API.lifecycleState: ENABLEDmulticlusteringressFeatureSpec:configMembership: projects/qwiklabs-gcp-6dbb2ce9d43a8ef1/locations/global/memberships/ingress-configname: projects/qwiklabs-gcp-6dbb2ce9d43a8ef1/locations/global/features/multiclusteringressupdateTime: ‘2020–03–23T15:55:36.087585371Z’

Press CTRL-C or CMD-C on a Macbook to exit the watch command.

Deploy Hipster Shop app to blue and green clusters

Deploy the Hipster Shop app to both blue and green clusters.

kubectl --context ${BLUE_CLUSTER} apply -f ${WORKDIR}/hipster-shop
kubectl --context ${GREEN_CLUSTER} apply -f ${WORKDIR}/hipster-shop

Wait a few minutes and ensure all Pods are Running in the blue and green clusters.

kubectl --context ${BLUE_CLUSTER} get pods
kubectl --context ${GREEN_CLUSTER} get podsOutput (do not copy)
NAME READY STATUS RESTARTS AGEadservice-86f5dfb995-nlm5w 1/1 Running 0 10mcartservice-76cf9686b6-rxf7b 1/1 Running 0 10mcheckoutservice-7766b946f5-qszvc 1/1 Running 0 10mcurrencyservice-76975c7847-vmwn7 1/1 Running 0 10memailservice-c55cd96f-74rxs 1/1 Running 0 10mfrontend-f4b7cd95-lk4k8 1/1 Running 0 10mloadgenerator-6784bc5f77-bkx4c 1/1 Running 0 10mpaymentservice-696f796545–8sjp5 1/1 Running 0 10mproductcatalogservice-7975f8588c-blrbq 1/1 Running 0 10mrecommendationservice-6d44459d79-xxb8w 1/1 Running 0 10mredis-cart-6448dcbdcc-8qcb4 1/1 Running 0 10mshippingservice-767f84b88c-8f26h 1/1 Running 0 10m

Configure multi cluster Ingress

In this section, you create a multicluster Ingress that sends traffic to the Hipster Shop frontend to both blue and green clusters. A Google Cloud Load Balancer (GCLB) is created which uses the frontend Services in both blue and green clusters as backends. To create the GCLB, you need two resources: a MultiClusterIngress and one or more MultiClusterServices. MultiClusterIngress and MultiClusterService objects are multi-cluster analogs to the existing Kubernetes Ingress and Service resources used in the single cluster context.

Deploy the MulticlusterIngress resource to the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mci.yamlOutput (do not copy)
multiclusteringress.networking.gke.io/frontend-multicluster-ingress created

Deploy the MulticlusterService resource to the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue-green.yamlOutput (do not copy)
multiclusterservice.networking.gke.io/frontend-multicluster-svc created

Inspect the MulticlusterIngress resource.

kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o yamlOutput excerpt (do not copy)
spec:template:spec:backend:serviceName: frontend-multicluster-svcservicePort: 80

MulticlusterIngress resource looks very much like the Kubernetes Ingress resource except that the serviceName spec points to a MulticlusterService.

Inspect the MulticlusterService resource.

kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o yamlOutput excerpt (do not copy)
spec:clusters:- link: us-west1-b/blue- link: us-west1-c/greentemplate:spec:ports:- name: webport: 80protocol: TCPtargetPort: 8080selector:app: frontend

MulticlusterService looks like a Kubernetes Service resource, except it has a clusters spec. The clusters value is the list of registered clusters where the MulticlusterService is created.

The MulticlusterIngress creates a GCLB in the project with a backend service pointing to the MulticlusterService. Wait until you see a public VIP for the GCLB. Run the following command until you see the GCLB VIP in the status.

watch kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o jsonpath=”{.items[].status.VIP}” Output (do not copy)
EXTERNAL IP ADDRESS

This step takes a few minutes. Do not proceed until you see a VIP assigned in the MulticlusterIngress status. Until the GCLB VIP is created, you will see null as the output for the command above.

Press CTRL-C or CMD-C on a Macbook to exit the watch command.
Access the Hipster Shop app by navigating to the IP address on the GCLB VIP from the previous step.

If you see a 404 error message, wait a bit longer and refresh the page until you see the Hipster shop frontend.

Browse through the app and make a few purchases. At this point the Hipster Shop app is deployed to both blue and green clusters and the MulticlusterIngress is set up to send traffic to both clusters.

Set up load generator

In this section, you set up a loadgenerator that generates client traffic to the GCLB VIP. Initially, this sends traffic to both blue and green clusters since the MulticlusterService is set up to send traffic to both clusters. Later, you configure the MulticlusterService to send traffic to a single cluster.

Get the GCLB VIP.

export GCLB_VIP=$(kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o json | jq -r ‘.items[].status.VIP’)
echo ${GCLB_VIP}Output (do not copy)
EXTERNAL IP ADDRES

Configure the loadgenerator to send client traffic to the GCLB VIP.

sed -i 's/GCLB_VIP/'${GCLB_VIP}'/g' ${WORKDIR}/load-generator/loadgenerator.yaml

Deploy the loadgenerator in the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f ${WORKDIR}/load-generator

Verify loadgenerator Pod is Running in the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} get podsOutput (do not copy)
NAME READY STATUS RESTARTS AGEloadgenerator-5498cbcb86-hqscp 1/1 Running 0 53sloadgenerator-5498cbcb86-m2z2z 1/1 Running 0 53sloadgenerator-5498cbcb86-p56qb 1/1 Running 0 53s

Monitor with Cloud Monitoring

In this section you monitor the traffic to the Hipster Shop app using Cloud Monitoring and Cloud Console.

Monitoring GCLB metrics with Cloud Console

In the previous section, you set up a loadgenerator deployment that simulates Hipster Shop client traffic by accessing it through the GCLB VIP. You can monitor GCLB metrics through the Cloud Console.

Get the name of the forwarding rule for the frontend-multicluster-ingress.

export INGRESS_LB_RULE=$(gcloud compute forwarding-rules list | grep frontend-multicluster-ingress | awk '{print $4}')echo ${INGRESS_LB_RULE}Output (do not copy)
mci-h8zu63-default-frontend-multicluster-ingress

Navigate to the link from the output of the command below.

echo "https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&tab=monitoring&duration=PT1H"Output (do not copy)
https://console.cloud.google.com/net-services/loadbalancing/details/http/mci-h8zu63-default-frontend-multicluster-ingress?project=gke-multicluster-upgrades&tab=monitoring&duration=PT1H

You can also get to this page via the Cloud Console by navigating to Network services > Load balancing page. Click on the forwarding rule that starts with ‘mci’ and then click on the Monitoring tab.

In the Load balancer details Monitoring page, select the backend beginning with mci from the Backend dropdown menu.

You can see that the traffic to the Hipster Shop app is going to both blue and green clusters (noted by the two zones the clusters are in). You can also see the timeline metrics chart showing traffic going to both backends.

It takes a few minutes for load balancer metrics to populate. Refresh the page after a few minutes if needed.

You can see the NEGs (under the Backend label starting with k8s1-) for the two frontend MulticlusterServices running in both the blue and green cluster.

Monitoring with Cloud Monitoring

You can monitor the metrics of your applications via Cloud Monitoring.

Navigate to the link from the output of the command below to access Cloud Monitoring metric explorer.

echo "https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D"

You can also access the same page via the Cloud Console. Click on Monitoring > Metrics explorer. In the Find resource type and metrics search bar, type and select `https_lb_rule`. For Select a metric, search for and select `Request count`. In the Group by search bar, search for and select `backend_scope`.

When accessing the link for the first time, Cloud Monitoring sets up a new workspace for you.

After the workspace is created, the chart shows the GCLB Request Count metrics grouped by backends.

You see traffic going to the two backends (blue and green clusters). The backends are represented by the zones for the two frontend MulticlusterServices running in both clusters.

You may see a third backend (named INVALID_BACKEND) for a short period of time. This is a UI anomaly in Cloud Monitoring and can be ignored.

Drain and upgrade the green cluster

In this section, you “drain” the green cluster. Draining a cluster means you remove it from the load balancing pool. After you drain the green cluster, all client traffic destined for the Hipster Shop app goes to the blue cluster. You can monitor this process as described in the previous section. Once the cluster is drained, you can upgrade the drained cluster. After the upgrade, you can put it back in the load balancing pool. You follow the same steps to upgrade the other cluster (not shown in this tutorial).

To drain the green cluster, you update the MulticlusterService resource in the ingress-cluster and remove the green cluster from the clusters spec.

Drain the green cluster

Update the MulticlusterService resource in the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue.yaml

Verify that you only have the blue cluster in the clusters spec.

kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o json | jq '.items[].spec.clusters'Output (do not copy)
{   “link”: “us-west1-b/blue”}

You see only the blue cluster in the clusters spec. This means only the blue cluster is in the GCLB load balancing pool.

Monitor Ingress metrics

Navigate to the Cloud Monitoring metrics chart page or by clicking the output of the following command.

echo "https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D"

The output shows that only the blue cluster is receiving traffic.

It may take a few minutes for Cloud Monitoring metrics to refresh.

You also see similar metrics from the GCLB metrics in the Cloud Console. Navigate to the GCLB Monitoring metrics page in Cloud Console or click on the output of the following command.

echo "https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&tab=monitoring&duration=PT1H"

The chart also shows only the blue cluster is receiving Hipster Shop traffic.

It may take a few minutes for load balancer metrics to refresh.

Upgrade the green cluster

Now that the green cluster is no longer receiving any client traffic, you can upgrade the cluster (master and nodes). Get the current cluster version for the master and the nodes. You can then find the available versions (in the zone). Finally, you can upgrade the cluster to the newer version.

Get the current version of the clusters.

gcloud container clusters listOutput (do not copy)
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUSingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 RUNNINGblue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 RUNNINGgreen us-west1-c 1.14.10-gke.24 35.197.46.200 n1-standard-1 1.14.10-gke.24 4 RUNNING

Your cluster versions may be different depending on the time you go through this tutorial.

Get the list of available master versions in the zone.

gcloud container get-server-config --zone us-west1-c --format=json | jq '.validMasterVersions'Output (do not copy)
[“1.15.9-gke.22”,“1.15.9-gke.12”,“1.15.9-gke.9”,“1.15.9-gke.8”,“1.15.8-gke.3”,“1.14.10-gke.27”,“1.14.10-gke.24”,“1.14.10-gke.22”,“1.14.10-gke.21”,“1.14.10-gke.17”,“1.13.12-gke.30”]

Get a list of available node versions in the zone.

gcloud container get-server-config --zone us-west1-c --format=json | jq '.validNodeVersions[0:20]'Output (do not copy)
[“1.15.9-gke.22”,“1.15.9-gke.12”,“1.15.9-gke.9”,“1.15.9-gke.8”,“1.15.8-gke.3”,“1.15.8-gke.2”,“1.15.7-gke.23”,“1.15.7-gke.2”,“1.15.4-gke.22”,“1.14.10-gke.27”,“1.14.10-gke.24”,“1.14.10-gke.22”,“1.14.10-gke.21”,“1.14.10-gke.17”,“1.14.10-gke.0”,“1.14.9-gke.23”,“1.14.9-gke.2”,“1.14.9-gke.0”,“1.14.8-gke.33”,“1.14.8-gke.21”]

Pick the desired master and node version (higher than the current master and node version) for the green cluster.

export UPGRADE_VERSION="1.15.9-gke.26"

Your cluster versions may be different depending on the time you go through this tutorial. In order to determine which versions to pick, refer to the upgrading clusters and node pools documentation. Generally, it is best practice to pick the same version for the master and the nodes. Pick one of the available versions in your list for both master and node.

Upgrade the master for the green cluster.

gcloud container clusters upgrade green \
 --zone us-west1-c --master --cluster-version ${UPGRADE_VERSION}

Type “Y” to confirm the upgrade.

Do you want to continue (Y/n)? YUpgrading green…⠧

This process takes a few minutes to complete. Please wait until the master upgrade is complete before proceeding. Once the update is complete, you should see the following message.
Updated [https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green].

Once the master upgrade is completed, you can upgrade the nodes in the green cluster by running the following command.

gcloud container clusters upgrade green \ --zone=us-west1-c --node-pool=default-pool \ --cluster-version ${UPGRADE_VERSION}

Type “Y” to confirm the upgrade.

Do you want to continue (Y/n)? YUpgrading green… Done with 0 out of 4 nodes (0.0%): 1 being processed…⠶

This process takes a few minutes to complete. Please wait until the node upgrade is complete before proceeding. After the upgrade is complete, you should see the following message.
Upgrading green… Done with 4 out of 4 nodes (100.0%): 4 succeeded…done.
Updated [https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green].

Verify that the green cluster (master and nodes) are upgraded.

gcloud container clusters listOutput (do not copy)
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUSingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 RUNNINGblue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 RUNNINGgreen us-west1-c 1.15.8-gke.3 35.197.46.200 n1-standard-1 1.15.8-gke.3 4 RUNNING

Your cluster versions may be different depending on the time you go through this tutorial.

Add green cluster to the load balancing pool

In this section, you add the green cluster back in the load balancing pool.

Verify that all Hipster Shop app deployments are running on the green cluster prior to adding it back to the load balancing pool.

kubectl --context ${GREEN_CLUSTER} get podsOutput (do not copy)
NAME READY STATUS RESTARTS AGEadservice-86f5dfb995–2b25h 1/1 Running 0 16mcartservice-76cf9686b6-ws7b7 1/1 Running 1 13mcheckoutservice-7766b946f5–6fhjh 1/1 Running 0 9m50scurrencyservice-76975c7847-rf8r7 1/1 Running 0 13memailservice-c55cd96f-pht8h 1/1 Running 0 13mfrontend-f4b7cd95-wxdsh 1/1 Running 0 13mloadgenerator-6784bc5f77–6b4cd 1/1 Running 6 6m34spaymentservice-696f796545–9wrl7 1/1 Running 0 9m49sproductcatalogservice-7975f8588c-kbm5k 1/1 Running 0 6m33srecommendationservice-6d44459d79-km8vm 1/1 Running 0 9m49sredis-cart-6448dcbdcc-sjg69 1/1 Running 0 13mshippingservice-767f84b88c-gh9m4 1/1 Running 0 9m49s

Update the MutliclusterService to add the green cluster back in the load balancing pool.

kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue-green.yaml

Verify that you have both blue and green clusters in the clusters spec.

kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o json | jq '.items[].spec.clusters'Output (do not copy)
{   “link”: “us-west1-b/blue”},{   “link”: “us-west1-c/green”}

You see both blue and green clusters in the clusters spec.

Monitor Ingress metrics

Navigate to the Cloud Monitoring metrics chart page or by clicking the output of the following command.

echo "https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D"

The output shows that both blue and green clusters are receiving traffic.

It may take a few minutes for Monitoring metrics to refresh.

You also see similar metrics from the GCLB metrics in the Cloud Console. Navigate to the GCLB Monitoring metrics page in Cloud Console or click on the output of the following command.

echo "https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&tab=monitoring&duration=PT1H"

The chart also shows both blue and green clusters are receiving Hipster Shop traffic.

It may take a few minutes for load balancer metrics to refresh.

Congratulations. You successfully upgraded a GKE cluster in a multi cluster architecture using Ingress for Anthos. You can upgrade the blue cluster using the same process.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete clusters

Unregister and delete blue and green clusters.

gcloud container hub memberships unregister green \ --project=${PROJECT} \ --gke-uri=${GREEN_URI}gcloud container clusters delete green --zone us-west1-c --quietgcloud container hub memberships unregister blue \ --project=${PROJECT} \ --gke-uri=${BLUE_URI}gcloud container clusters delete blue --zone us-west1-b --quiet

Delete the MuticlusterIngress from the ingress-config cluster. This deletes the GCLB resources from the project. Unregister and delete the ingress-config cluster.

kubectl --context ${INGRESS_CONFIG_CLUSTER} delete -f mci.yaml
gcloud container hub memberships unregister ingress-config \ --project=${PROJECT} \ --gke-uri=${INGRESS_CONFIG_URI}gcloud container clusters delete ingress-config --zone us-west1-a --quiet

Verify all clusters are deleted.

gcloud container clusters listOutput (do not copy)
<null>

Unset the kubeconfig file. This resets to the default kubeconfig file.

unset KUBECONFIG

Remove the WORKDIR folder.

cd ${HOME}
rm -rf ${WORKDIR}

Delete the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

In the Cloud Platform Console, go to the Projects page.

GO TO THE PROJECTS PAGE

In the project list, select the project you want to delete and click Delete.

In the dialog, type the project ID, and then click Shut down to delete the project.

What’s next

Learn more about Ingress for Anthos
Learn how to deploy Ingress for Anthos — Deploying Ingress for Anthos across clusters
Try out other Google Cloud Platform features for yourself. Have a look at our tutorials.

Kubernetes Engine (GKE) multi-cluster life cycle management series

Part V: GKE multi-cluster upgrades using Ingress for Anthos step-by-step tutorial

Prerequisites

Topology

Objectives

Costs

Before you begin

Set up the environment

Create and register GKE clusters to GKE Hub

Create GKE clusters

Register GKE clusters to GKE Hub

Deploy Hipster Shop app to blue and green clusters

Configure multi cluster Ingress

Set up load generator

Monitor with Cloud Monitoring

Monitoring GCLB metrics with Cloud Console

Monitoring with Cloud Monitoring

Drain and upgrade the green cluster

Drain the green cluster

Monitor Ingress metrics

Upgrade the green cluster

Add green cluster to the load balancing pool

Monitor Ingress metrics

Cleaning up

Delete clusters

Delete the project

What’s next

Written by Ameer Abbas