Kubernetes Multi-Cluster deployments with GKE: Part 2 -Service Export

Introduction

In the earlier post Part 1, we discussed on the need for Multi cluster architecture in GKE , the advantages and challenges involved in this architecture. We also looked at the concept of Kubernetes Fleet and how Fleet enables to group multiple clusters under a common Fleet Entity. Refer the GKE documentation on “Register clusters to the Fleet” to read in detail about Fleet and understand how to registers multiple clusters to a Fleet.

In this part, we will discuss on how to achieve east west communication among different clusters. This enables inter-cluster communication of different services across clusters registered to a common “Fleet”. We will also look at some sample deployment topologies where a multi cluster GKE set up with east-west communication can be enabled.

Multi-cluster services with ServiceExport

To address the problem of east-west communication across multiple-clusters, GKE has introduced a new Custom Resource called ServiceExport. The “Service Export” Resource specifies the name of the service that will be exposed outside the current cluster. Hence it is possible to pick and chose a specific subset of Kubernetes Services to be exposed outside the current Cluster.

As shown in the figure above, the service 2 in cluster 2 can be exported, in which case it is visible to another cluster (cluster 1 in the fleet) . The pod1 in cluster 1 can invoke this service via Service DNS name just like how it invokes another service running within the same cluster.

Note: The ServiceExport resources exports the service to be visible only across the clusters registered to the Fleet. The service is not visible to clusters that are not part of the Fleet. Also for the Service Export to be visible it is essential that the namespace should exist in all the clusters across the fleet.

An example of Service Export resource is as below

kind: ServiceExport

apiVersion: net.gke.io/v1

metadata:

namespace: NAMESPACE

name: SERVICE_NAME

In the “service export” resource, we specify the name of the service and the namespace of the service to be exported.

The Service Export object is visible across all clusters with the following DNS name:

SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local

Notice that it ends with clusterset.local instead of the usual cluster.local. Any cluster within the fleet can access this “exported service” residing in a different cluster with this DNS name.

Behind the scenes , the GKE operator/controller listens for a Service Export object to be created. Once a “serviceexport” is created the GKE serviceexport controller creates an implicit “Service Import” Object in all the clusters belonging to this Fleet. You can verify this by running the command “kubectl get serviceexport” and kubectl get serviceimport

The Service import object that is implicitly created across all the fleets in the cluster is as below

Note: The Service Import object is not created by the user. It is created by mcs controller automatically when a Service Export resource is created to expose a service.

To Summarise, you can export any service within a cluster to make the service available to all clusters in the Fleet. This enables the service to be invoked by pods running in different clusters. Hence services running across cluster boundaries(within the same Fleet of course) are able to talk to each other.

Use Case

There are multiple use-cases where Service Export feature is immensely valuable. Let’s consider a classic use case where service export with multiple clusters can be immensely valuable. Lets consider a typical enterprise application consisting of micro-services running in Kubernetes, developed by individual teams(A, B and C). These services running as Kubernetes pods communicate with each other to serve the business functionality.

In this scenario as illustrated in the below figure, each team can deploy their application to individual GKE clusters. These clusters can in fact be part of different projects, so that each teams have completely independent resource boundaries. With cluster per team architecture, individual teams can define their IAM policies, cluster configurations, network configurations independent of each other. In this model every team creates and manage their own GKE cluster.

Since GKE enables communication between services across cluster boundaries via Service Export and Import, the services deployed in different clusters are able to talk to each other.

Note: These clusters are registered to a common Fleet as shown above.

So essentially GKE Service export and Service import feature supports the flexibility, security and independence of a cluster per team architecture by enabling these services to communicate across cluster boundaries.

Conclusion

With “ServiceExport” feature, GKE enables enterprises to adopt the flexibility offered by multiple clusters. So different teams or BUs within an organisation can create , manage their individual clusters and deploy their services. Though these micro-services are deployed across clusters they can still communicate with each other enabling inter-service communication across clusters.