Istio multicluster with multiple network using istio-gateway

Goal

Enable communication between istio clusters, istio-cluster 1 and istio-cluster2, located on separate networks.

Prerequisite

Prior to reading this article, it is recommended to read the article on Istio multicluster with istio-csr + cert-manager + vault PKI to deploy istio multicluster.

Architecture

multicluster (multi primary) multiple network

Step 1 set default network

add label topology.istio.io/network=network1 on namespace istio-system for istio-cluster1

kubectl label namespace istio-system topology.istio.io/network=network1

and topology.istio.io/network=network2 on namespace istio-system for istio-cluster2

kubectl label namespace istio-system topology.istio.io/network=network2

When this label is applied to the system namespace (istio-system). it sets up a default network for pods under the control plane’s management.

Step 2 change network istio

in article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 6 Deploy istio point 6 change line 474 for istio-cluster1:

network: "network1"

and istio-cluster2:

network: "network2"

Step 3 Deploy istio-gateway

step 1. change line 95 on both cluster to enable port:

• 15443 TLS SNI Routing
• 15012 GRPC XDS and CA services (TLS and mTLS, recommended for production use)
• 15017 HTTPS for Webhook container port, forwarded from 443

and annotation ISTIO_META_REQUESTED_NETWORK_VIEW=<network>indicates that traffic through this gateway should be routed inside the network below:

istio-cluster1

networkGateway: "network1"

istio-cluster2

networkGateway: "network2"

step 2. add new label below line 81 on both cluster for selector gateway istio to:

labels:
  istio: eastwestgateway

step 3. deploy istio-gateway

helm upgrade --install istio-gateway --version 1.20.0 --values values.yaml -n istio-system istio/gateway

Step 4 expose services

expose custom port 15443 istio-gateway on both cluster

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

tls.mode AUTO_PASSTHROUGH Similar to the passthrough mode, except servers with this TLS mode do not require an associated VirtualService to map from the SNI value to service in the registry. ref

Port 15443 istio for SNI Routing. Server Name Indication (SNI) routing serves as an extra means to route HTTPS or other TLS-based protocols. With SNI, traffic can be directed to a specific destination without needing SSL Termination.

Step 5 Configuring istio multicluster

note: this step same with on article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 7 Configuring istio multicluster

step 1. install istioctl

step 2. create remote secret from istio-cluster2. change <your istio-cluster1 context> and <your istio-cluster2 context> to your related context.

istioctl create-remote-secret \
    --context="<your istio-cluster1 context" \
    --name=istio-cluster1 | \
    kubectl apply -f - --context="<your istio-cluster2 context"

step 3. create remote secret from istio-cluster1. change <your istio-cluster1 context> and <your istio-cluster2 context> to your related context.

istioctl create-remote-secret \
    --context="<your istio-cluster2 context" \
    --name=istio-cluster2 | \
    kubectl apply -f - --context="<your istio-cluster1 context"

Step 6 Testing istio multicluster

note: this step same with on article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 8 Testing Istio Multicluster

step 1. create namespace sample with injected istio sidecar on both clusters.

kubectl create ns sample
kubectl label ns sample istio-injection=enabled

step 2. Create HelloWorld service in both clusters

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l service=helloworld -n sample

step 3. Deploy HelloWorld V1 apps to istio-cluster1

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v1 -n sample

step 4. Deploy HelloWorld V2 apps to istio-cluster2

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v2 -n sample

step 5. Deploy sleep apps to both clusters

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml \
-n sample

step 6. Send multiple request from sleep pod to helloworld service. test to both clusters.

for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld:5000/hello; done

make sure that the helloworld version should toggle between v1 and v2

Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq

step 7. From each cluster, find the endpoints the sleep service has for helloworld:

$ istioctl proxy-config endpoint sleep-dd98b5f48-djwdw.sample | grep helloworld

In multi-network, we expect one of the endpoint IPs to match the remote cluster’s east-west gateway public IP:

34.124.191.238:15443                                    HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

conclusion

1. In a multiple network mode, istio-gateway is essential for inter-cluster communication since direct connection via pod IP addresses isn’t feasible.
2. it’s crucial to recognize that each cluster operates within its own distinct network environment.
3. Communication between pods across clusters through istio-gateway.
4. port 15443 on istio-gateway is utilized for SNI Routing.