Google Cloud Anthos Series -Anthos Multi-cluster Ingress
Google Cloud Anthos Series : Part-6
Welcome to Part-6 of the “Google Cloud Anthos Series”. You can find the complete series here
Multi Cluster Ingress is a Google-hosted service that helps customers build resilient Anthos architecture through clusters deployed across multiple cloud regions. In addition to resiliency it also helps redirect traffic to the nearest cluster to ensure lowest latency and consistent user experience.Under the wraps, the service leverages Google Front Ends(GFEs) through the External HTTP(s) load balancing service. Any incoming request is routed by GFEs to the GKE cluster closest to the client. Organizations can also choose multi cluster architecture with multi cluster ingress when there are requirements like regional isolation of workloads or data residency
Multi Cluster Ingress Architecture components
Multi-cluster Ingress architecture components and their features are listed below in brief
Multi Cluster Ingress controller(Anthos Ingress Controller)
- Refers to a control plane outside of the member clusters
- Leverages External Load balancer of GCP by creating backend Network Endpoint Groups (NEGs)
- Backend endpoints of the LB dynamically track pod endpoints
- Tracks pods in clusters and ensures that the load balancer is aware of updates in the cluster
Config cluster
- Used to deploy custom resource types called MultiClusterIngress and MultiClusterService required for Multi cluster Ingress operations
- One GKE cluster is designated as the control plane that centrally managed the multi cluster ingress resources
- MultiClusterService(MCS) represents a service deployed across different cluster that will be exposed by the load balancer.
- MultiClusterIngress(MCI) is similar to ingress resources in k8s cluster that defined backends, paths, protocols etc that sends matching traffic to MCS
Fleets
- Formerly known as environ
- Groups together multiple clusters that will act as backend for Multi Cluster Ingress
- Supports ‘namespace sameness’ ie resources with same name and same namespace across multiple clusters are grouped as single workload
Member cluster
- Refers to GKE clusters registered to a fleet
- Hosts the workloads to which Multi Cluster ingress redirects traffic
- MCS by default sends traffic to all backend clusters, but this can be customised in MCS config yaml to target specific clusters
High level architecture of Multi cluster ingress to multiple GKE clusters is as shown below:
Multi Cluster Ingress deployment
Before we start with the deployment, a word on the pricing. While using only multi cluster Ingress service with GKE clusters, customers can opt for a standalone pricing. This pricing get applied only if Anthos API is disabled. If Anthos API is enabled and you are using other features of Anthos as well, then the billing will depend on the Cluster vCPUs and Anthos Pricing. For this deployment we will be using the Anthos pricing and the Online Boutique application we had used in our earlier Google Cloud Devops blog series
Note: To understand more about Anthos pricing, you can refer to part 3 of this blog series
Deployment considerations
- Steps to deploy Multi Cluster ingress requires GKE administrator rights on the GCP project as it requires elevated permissions
- It is recommended to use workload identity for clusters for enabling seemless authentication for workloads in cluster while accessing other GCP resources
Diagram below shows Samajik’s end state architecture using Multi cluster Ingress
Step by step deployment of Multi Cluster Ingress from Cloud shell is given below. Replace variables in the code marked in bold as applicable with values specific to your environment ie project id, cluster name, GCP region etc
- Enable APIs required for the deployment . The below command should be run to use Anthos pricing
gcloud services enable \
anthos.googleapis.com \
multiclusteringress.googleapis.com \
gkehub.googleapis.com \
container.googleapis.com \
--project=<<Your-project-id>>
2. Deploy clusters with workload identity enabled. Here we are deploying three clusters named cluster1, cluster2 and cluster3 across three different regions
gcloud container clusters create cluster1 \
--zone=us-central1-b \
--enable-ip-alias \
--workload-pool=<<Your-project-id>>.svc.id.goog \
--release-channel=stable \
--project=<<Your-project-id>>gcloud container clusters create cluster2 \
--zone=europe-west1-b \
--enable-ip-alias \
--workload-pool=<<Your-project-id>>.svc.id.goog \
--release-channel=stable \
--project=<<Your-project-id>>gcloud container clusters create cluster3 \
--zone=asia-east1-b \
--enable-ip-alias \
--workload-pool=<<Your-project-id>>.svc.id.goog \
--release-channel=stable \
--project=<<Your-project-id>>
3. Retrieve the credentials for the clusters and rename them for ease of use. Repeat the below steps for all the three clusters after replacing the cluster name
gcloud container clusters get-credentials cluster1 \
--zone=us-central1-b \
--project=<<Your-project-id>>kubectl config rename-context gke_<<Your-project-id>>_us-central1-b_gcluster1 cluster1
4. Add clusters to the fleet using workload identity. Repeat the below command for all three clusters after replacing the cluster name
gcloud container hub memberships register cluster1 \
--gke-cluster us-central1-b/cluster1 \
--enable-workload-identity \
--project=<<Your-project-id>>
5. Confirm fleet membership using the below command
gcloud container hub memberships list --project=<<Your-project-id>>
Output will be similar to below
6. Configure cluster1 as the config cluster
gcloud beta container hub ingress enable \
--config-membership=cluster1
7. Create a yaml file namespace.yaml that defines the namespace using the below content
apiVersion: v1
kind: Namespace
metadata:
name: mcinamespace
8. Switch context to cluster1 and deploy namespace.yaml to create the namespace. Repeat the same steps for cluster2 and cluster3
kubectl config use-context cluster1
kubectl apply -f namespace.yaml
9. Clone the online boutique microservices demo application from GitHub
git clone https://github.com/GoogleCloudPlatform/microservices-demo.git
cd microservices-demo/release/
10. Edit the kubernetes-manifest. yaml and remove the following block of code that configures load balancer ingress . We will be configuring Multi Cluster Ingress for the frontend service later
— -
apiVersion: v1
kind: Service
metadata:
name: frontend-external
spec:
type: LoadBalancer
selector:
app: frontend
ports:
— name: http
port: 80
targetPort: 8080
11. Deploy the application in all three clusters by switching the context. The below commands will create the deployment in cluster 1. Repeat the same for cluster 2 and cluster 3
kubectl config use-context cluster1kubectl apply -f kubernetes-manifests.yaml -n mcinamespace
12. Once deployed successfully you can see the workloads listed in the portal for each cluster
Note: If you see an error in scheduling pods due to insufficient resources, enable autoscaling or create additional node pools to schedule the pods.
13. Next step is to create a MCS resource that can represent the frontend service across all three clusters. Create a file mcs_boutique.yaml with the below content.
apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
name: boutique-mcs
namespace: mcinamespace
spec:
template:
spec:
selector:
app: frontend
ports:
- name: web
protocol: TCP
port: 8080
targetPort: 8080
14. Run the below commands only in the config cluster ie cluster 1 to create MCS
kubectl config use-context cluster1kubectl apply -f mcs_boutique.yaml
15.Run the below command in other clusters after switching context and you can see that an associated headless service is created in all clusters that have the pods with label app:mcinamespace
kubectl config use-context cluster2
kubectl get service -n mcinamespace
16. Create a file mci_boutique.yaml using the below content. It will be used to create the MCI service
apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
name: mci-ingress
namespace: mcinamespace
spec:
template:
spec:
backend:
serviceName: boutique-mcs
servicePort: 8080
17.Run the below commands only in the config cluster ie cluster 1 to create MCI
kubectl config use-context cluster1
kubectl apply -f mci_boutique.yaml
18. Check status of the MCI service using the below command . You should see a VIP created for the service
kubectl describe mci mci-ingress -n mcinamespace
In addition to VIP you can also see that there are multiple resources created by the MCI controller ie load balancer, backend service, firewall rule, forwarding rule, Network Endpoint Group and health checks. The load balancer can be seen from the Google cloud console as well .
The frontend IP will be same as the VIP shown earlier. You can see that the backend is configured to distribute traffic across NEGs pointing to the three member clusters in different regions
19. The url can be accessed at http://MCI VIP> or http://<loadbalancer frontendIP>
Coming up..
Samajik’s requirement of effortlessly growing their services across several cloud locations is addressed by Multi Cluster Ingress.Guhan is now even more excited to explore other features of Anthos that Samajik can benefit from. Stay tuned to Guhan and Ram’s conversation to learn more…
Contributors : Pushkar Kothavade, Anchit Nishant, Dhandus