Gateway | GKE Multi-tenancy: Creating “One per Cluster” shared Load Balancer for Multi-tenant GKE cluster using Gateway API

In multi-tenancy, a cluster team administers the kubernetes cluster and configures appropriate access for each of the application teams (tenants), who share the cluster for their deployments.

Now, when it comes to exposing the applications (k8s services) to the outside world, the process involves two significant steps.

The first is, specification of a Loadbalancer, with details such as type of Loadbalancer, ports and protocols to listen on etc. This represents the Frontend configuration of a Loadbalancer.

The second is, specification of rules for routing incoming requests, with details such as which traffic the routes should match and which services to route the requests to etc. This represents the Routing configuration of a Loadbalancer.

Between the two steps mentioned above, the cluster administrators would want to own the first one, while giving application teams(tenants) the necessary access to perform the second one. This way, cluster administrators define how shared infrastructure can be used by many different and non-coordinating application teams.

Ingress VS Gateway

Ingress combines specification of Loadbalancer Frontend and Routing configurations (both of the steps above) into one single “Ingress” object, while Gateway API allows those to be separated.

Using the Gateway API,

a “Gateway” resource can be used for the Loadbalancer Frontend config and an “HTTPRoute” resource can be used for Routing config, clearly achieving the separation required for multi-tenancy.

Now, Let’s look at implementation with the help of sample configurations

GKE Multi-tenancy: Shared Gateway per cluster

As shown in the diagram above, we will assume the following namespaces and provision Gateway API resources in each of those namespaces respectively.

Infrastructure related:

infra-ns : a namespace chosen for shared infrastructure related resources, with cluster admin only access

This namespace will have a Gateway resource created to share with tenant-specific namespaces

Tenants related:

accounts-ns : a namespace chosen for a tenant team (assuming accounts related application team as a tenant), with accounts developer access

This namespace will have HTTPRoute resources created (along with associated app deployments and services), referencing the shared Gateway resource defined within infra-ns namespace as a parent

analytics-ns : Namespace for a second tenant. Implementation for this will be similar to the accounts-ns namespace (starts from Step 3).

Note: Make sure to enable Gateway API before using Gateway resources in GKE. You can enable the Gateway API on a new or existing GKE Autopilot and Standard clusters.

Create RBAC config for cluster administrators and provision Gateway resource

Step 1: Create RBAC policy for managing Gateway

For the cluster admin RBAC configuration, create a ClusterRole with rules to access Gateway resources and a ClusterRoleBinding to grant the permissions to cluster admin specific Groups/ Users

Use a sample RBAC config for Cluster Admin Group, as follows

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-gateway-admin-role
rules:
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["gateways"]
  verbs: ["get", "list", "create", "update", "patch", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-gateway-admin-role-binding
subjects:
- kind: Group
  name: cluster-gateway-admin-group@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-gateway-admin-role
  apiGroup: rbac.authorization.k8s.io

With the above config in-place, all users part of “cluster-gateway-admin-group” will have access to create and manage lifecycle of Gateway resources

Step 2: Create Gateway resource

The next step is to create a Gateway resource. For this you can use the following sample config

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: infra-ns
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway-access: "true"

The definition of this Gateway resource specifies the following

GatewayClassName: This field specifies the type of Load Balancer to be provisioned. The gke-l7-rilb specifies “Internal Application Load Balancer(s) built on the IALB” (For more, visit GatewayClass)

Listeners: Specifies the protocol and port configuration

AllowedRoutes: Specifies rules for where the HttpRoute resources can be linked from. In this case, it specifies the namespaces with label shared-gateway-access: “true” as eligible.

Also note that, this Gateway resource is created in a namespace called infra-ns (a namespace chosen for shared infrastructure related resources). This namespace will be used to link the HttpRoute resources with Gateway

Creation of a HttpRoute resource with with RBAC config for application team (tenant)

Step 3: Create tenant namespace and RBAC policies for managing HTTPRoute

In this step, create RBAC config necessary for application teams (tenants) and provision HttpRoute resource

Earlier, we assumed accounts-ns as a tenant namespace, which will be used here. This namespace should be labeled as shared-gateway-access: “true” based on the Gateway definition AllowedRoutes configuration mentioned in Step 2 (as shown below)

apiVersion: v1
kind: Namespace
metadata:
  name: accounts-ns
  labels:
    shared-gateway-access: "true"

For the application team’s RBAC configuration, create a Role with rules to access HttpRoute resources and a RoleBinding to grant the permissions to application team specific Groups/ Users

Use the sample RBAC config for Accounts Application Dev Group, as follows

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: accounts-ns
  name: accounts-ns-rw
rules:
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["httproutes"]
  verbs: ["get", "list", "create", "update", "patch", "delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: accounts-ns
  name: accounts-ns-rw-binding
subjects:
- kind: Group
  name: accounts-app-dev-group@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: accounts-ns-rw
  apiGroup: rbac.authorization.k8s.io

With the above config in-place, all users part of “accounts-app-dev-group” will have access to create HttpRoute resources within accounts-ns namespace

Step 4: Create Deployment and Service in tenant namespace

The next step is to create an HTTPRoute resource. Since it requires a k8s service, let’s use a sample deployment and service specs as follows

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: accounts-ns
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: accounts-service
  namespace: accounts-ns
spec:
  ports:
  - name: host1
    port: 80
    targetPort: 80
  selector:
    app: nginx
  type: ClusterIP

Note: For creating Deployment and Service objects within accounts-ns namespace “accounts-app-dev-group” will also require respective permissions through RBAC policies(which are not covered here) . You can define a Role separately (similar to the one defined in Step 3) for granting access to “pods”, “services”, “deployments” resources.

Step 5: Create HTTPRoute in tenant namespace

Now, let’s expose the nginx-service we created in the previous step, using HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: accounts-app
  namespace: accounts-ns
spec:
  parentRefs:
  - name: shared-gateway
    namespace: infra-ns
  rules:
  - matches:
    - path:
        value: /
    backendRefs:
    - name: accounts-service
      port: 80

The definition of this HTTPRoute resource specifies the following

ParentRefs: Referencing the Gateway resource from infra-ns namespace, which was created with name shared-gateway . Also, note that the namespace accounts-ns is labeled as shared-gateway-access: “true”. This reference binds HttpRoute with Gateway.

Rules: Specify rules for traffic matching with path values. Also, declare the backend services to route the matching requests to, with the name of k8s service name and port.

With this, implementation is done for “accounts” tenant. Repeat steps from 3 to 5 to configure it for additional tenants.

And that’s it! This final step concludes the successful configuration of “One per Cluster” shared Load Balancer for Multi-tenant GKE cluster using Gateway API.

Conclusion

The Gateway API, with its role-oriented resource model for Kubernetes service networking allows shared network infrastructure to be used by cluster tenants based on the policies set by cluster administrators.

We have seen steps involved in this process for setting up a single, shared Internal Application/HTTP(S) Loadbalancer, as an example.

This implementation can be extended for both Internal and External Loadbalancers, by having two Gateway resources created, to handle each for internal and external facing applications respectively.

References

Gateway | Google Kubernetes Engine (GKE) | Google Cloud

Introduction