Istio multicluster with multiple network using istio-gateway
Goal
Enable communication between istio clusters, istio-cluster 1 and istio-cluster2, located on separate networks.
Prerequisite
Prior to reading this article, it is recommended to read the article on Istio multicluster with istio-csr + cert-manager + vault PKI to deploy istio multicluster.
Architecture
multicluster (multi primary) multiple network
Step 1 set default network
add label topology.istio.io/network=network1 on namespace istio-system for istio-cluster1
kubectl label namespace istio-system topology.istio.io/network=network1
and topology.istio.io/network=network2 on namespace istio-system for istio-cluster2
kubectl label namespace istio-system topology.istio.io/network=network2
When this label is applied to the system namespace (istio-system). it sets up a default network for pods under the control plane’s management.
Step 2 change network istio
in article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 6 Deploy istio point 6 change line 474 for istio-cluster1:
network: "network1"
and istio-cluster2:
network: "network2"
Step 3 Deploy istio-gateway
step 1. change line 95 on both cluster to enable port:
- 15443 TLS SNI Routing
- 15012 GRPC XDS and CA services (TLS and mTLS, recommended for production use)
- 15017 HTTPS for Webhook container port, forwarded from 443
and annotation ISTIO_META_REQUESTED_NETWORK_VIEW=<network>
indicates that traffic through this gateway should be routed inside the network below:
istio-cluster1
networkGateway: "network1"
istio-cluster2
networkGateway: "network2"
step 2. add new label below line 81 on both cluster for selector gateway istio to:
labels:
istio: eastwestgateway
step 3. deploy istio-gateway
helm upgrade --install istio-gateway --version 1.20.0 --values values.yaml -n istio-system istio/gateway
Step 4 expose services
expose custom port 15443 istio-gateway on both cluster
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: cross-network-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
number: 15443
name: tls
protocol: TLS
tls:
mode: AUTO_PASSTHROUGH
hosts:
- "*.local"
tls.mode AUTO_PASSTHROUGH Similar to the passthrough mode, except servers with this TLS mode do not require an associated VirtualService to map from the SNI value to service in the registry. ref
Port 15443 istio for SNI Routing. Server Name Indication (SNI) routing serves as an extra means to route HTTPS or other TLS-based protocols. With SNI, traffic can be directed to a specific destination without needing SSL Termination.
Step 5 Configuring istio multicluster
note: this step same with on article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 7 Configuring istio multicluster
step 1. install istioctl
step 2. create remote secret from istio-cluster2. change <your istio-cluster1 context>
and <your istio-cluster2 context>
to your related context.
istioctl create-remote-secret \
--context="<your istio-cluster1 context" \
--name=istio-cluster1 | \
kubectl apply -f - --context="<your istio-cluster2 context"
step 3. create remote secret from istio-cluster1. change <your istio-cluster1 context>
and <your istio-cluster2 context>
to your related context.
istioctl create-remote-secret \
--context="<your istio-cluster2 context" \
--name=istio-cluster2 | \
kubectl apply -f - --context="<your istio-cluster1 context"
Step 6 Testing istio multicluster
note: this step same with on article Istio multicluster with istio-csr + cert-manager + vault PKI on Step 8 Testing Istio Multicluster
step 1. create namespace sample
with injected istio sidecar on both clusters.
kubectl create ns sample
kubectl label ns sample istio-injection=enabled
step 2. Create HelloWorld service in both clusters
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l service=helloworld -n sample
step 3. Deploy HelloWorld V1 apps to istio-cluster1
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v1 -n sample
step 4. Deploy HelloWorld V2 apps to istio-cluster2
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v2 -n sample
step 5. Deploy sleep apps to both clusters
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml \
-n sample
step 6. Send multiple request from sleep pod to helloworld service. test to both clusters.
for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld:5000/hello; done
make sure that the helloworld version should toggle between v1 and v2
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
step 7. From each cluster, find the endpoints the sleep
service has for helloworld
:
$ istioctl proxy-config endpoint sleep-dd98b5f48-djwdw.sample | grep helloworld
In multi-network, we expect one of the endpoint IPs to match the remote cluster’s east-west gateway public IP:
34.124.191.238:15443 HEALTHY OK outbound|5000||helloworld.sample.svc.cluster.local
conclusion
- In a multiple network mode, istio-gateway is essential for inter-cluster communication since direct connection via pod IP addresses isn’t feasible.
- it’s crucial to recognize that each cluster operates within its own distinct network environment.
- Communication between pods across clusters through istio-gateway.
- port 15443 on istio-gateway is utilized for SNI Routing.