Setup a multi-cluster environment with Istio and Kiali (Experimental support)

Kiali v1.69 includes experimental support for multi-cluster, which allows to have Kiali with Istio Mesh installed across multiple Kubernetes clusters. It is now possible to visualize multiple workloads in the same graph from different clusters, as well as the workload’s health, the services, the applications and the Istio configurations, from the same Kiali instance.

Further details can be seen in the multi-cluster section in Kiali.io. In this article we will describe how to deploy a multi-cluster environment, install workloads distributed in different clusters, and how to set up Kiali to visualize the multi-cluster mesh from a single Kiali instance:

• Requirements
• Installation
• Demo

Requirements

The experimental support indicates limited support for multi-cluster. At this first stage, just the primary-remote deployment model is supported. Also, Kiali will not be able to write (i.e. create/modify/delete) any resource in the remote clusters. Regarding authentication strategies, just anonymous authentication is supported. OpenID authentication is supported, but this support might depend on the provider. The purpose of this experimental release is to receive early feedback for the next iteration to GA status.

Istio supports different deployment models. The primary-remote model supports a single control plane and one or more data planes, across more than one cluster. A single Kiali will be deployed in the primary cluster.

Kiali in a primary-remote deployment model

Kiali requires a single, aggregated metrics endpoint that unifies the mesh metrics. The same is true for traces. The unified datastores should be accessible from the primary cluster. The mechanism for producing the unified store is outside the scope of Kiali. In this example, the metrics will be unified using Prometheus federation.

Kiali will identify the remote clusters by reading the Kiali remote cluster secrets that were created at Kiali installation time. The remote secrets consist of a service account, a role and a role binding in the remote cluster, and a secret with the remote service account credentials in a kubeconfig file that it is mounted in the Kiali pod. This file also indicates the name of the remote cluster.

There is a Kiali configuration item to define the cluster name. By default, the cluster name will be obtained from the Istio configuration. To avoid the auto-detection, as it may fail if istiod is not available when Kiali starts, it can be explicitly defined in the Kiali configuration. It is important both names match, so the remote cluster can be found in the telemetry queries.

Kiali will communicate with the remote clusters through the Kubernetes API server. The Istio CRDs don’t need to be installed in the remote clusters.

The supported authentication strategies are anonymous and OpenID (With the restrictions mentioned earlier). In this example we will use two minikube clusters with the anonymous authentication strategy.

How to install Kiali in a multi-cluster environment

The following environment will be installed:

• Primary cluster: east
• Remote cluster: west
• Minikube clusters
• Anonymous authentication

Requirements:

• Minikube (Tested in 1.30.1)
• kubectl (Tested with 1.27)
• istioctl (Tested in 17.2)

Export the cluster names:

export CTX_CLUSTER1=east
export CTX_CLUSTER2=west

Export the Istio directory to install the addons:

export ISTIO_DIR=~/bin/istio-1.17.2

Create Minikube using the Kiali hack script. This script will create the Minikube profile and enable ingress, the image registry and metallb load balancer addon:

curl -L -o k8s-minikube.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/k8s-minikube.sh
chmod +x k8s-minikube.sh

./k8s-minikube.sh start -lba "20-30" -mp $CTX_CLUSTER1
./k8s-minikube.sh start -lba "40-50" -mp $CTX_CLUSTER2

Create a self-signed certificate to setup in both clusters to be able to communicate:

mkdir -p /tmp/certs
pushd /tmp/certs

make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk root-ca

make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk cluster2-cacerts

kubectl create namespace istio-system --context $CTX_CLUSTER1
kubectl create secret generic cacerts -n istio-system --context $CTX_CLUSTER1 \
      --from-file=cluster1/ca-cert.pem \
      --from-file=cluster1/ca-key.pem \
      --from-file=cluster1/root-cert.pem \
      --from-file=cluster1/cert-chain.pem

kubectl create namespace istio-system --context $CTX_CLUSTER2
kubectl create secret generic cacerts -n istio-system --context $CTX_CLUSTER2 \
      --from-file=cluster2/ca-cert.pem \
      --from-file=cluster2/ca-key.pem \
      --from-file=cluster2/root-cert.pem \
      --from-file=cluster2/cert-chain.pem

popd

Setup the network for cluster east and install Istio and addons:

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: east
      network: network1
EOF

istioctl install --skip-confirmation=true --set hub=gcr.io/istio-release --set values.meshConfig.enableAutoMtls=true --set values.gateways.istio-egressgateway.enabled=true --set values.gateways.istio-ingressgateway.enabled=true --set values.global.meshID=mesh1 --set values.global.multiCluster.clusterName=east --set values.global.network=network1 --set values.meshConfig.accessLogFile=/dev/stdout --set values.pilot.env.EXTERNAL_ISTIOD=true --context="${CTX_CLUSTER1}" -f cluster1.yaml

kubectl apply -f ${ISTIO_DIR}/samples/addons/prometheus.yaml -n istio-system --context="${CTX_CLUSTER1}"
kubectl apply -f ${ISTIO_DIR}/samples/addons/grafana.yaml -n istio-system --context="${CTX_CLUSTER1}"
kubectl apply -f ${ISTIO_DIR}/samples/addons/jaeger.yaml -n istio-system --context="${CTX_CLUSTER1}"

Generate the east gateway, and expose Istiod and the services:

${ISTIO_DIR}/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster ${CTX_CLUSTER1} --network network1 | istioctl --context="${CTX_CLUSTER1}" install -y -f -

kubectl apply --context=${CTX_CLUSTER1} -n istio-system -f ${ISTIO_DIR}/samples/multicluster/expose-istiod.yaml
kubectl apply --context=${CTX_CLUSTER1} -n istio-system -f ${ISTIO_DIR}/samples/multicluster/expose-services.yaml

Prepare the remote secrets to be used by Kiali:

curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh
chmod +x kiali-prepare-remote-cluster.sh
./kiali-prepare-remote-cluster.sh --kiali-cluster-context ${CTX_CLUSTER1} --remote-cluster-context ${CTX_CLUSTER2}

The previous command is using a helper script that creates a Kiali Service account, a role and a role binding in the remote cluster, and then creates a kubeconfig file to save it as a secret so Kiali can read it.

Install Kiali:

kubectl config use-context ${CTX_CLUSTER1} 
helm upgrade --install --namespace istio-system --set kubernetes_config.cache_enabled=false --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server 

Setup the network for cluster west:

kubectl --context=${CTX_CLUSTER2} annotate namespace istio-system topology.istio.io/controlPlaneClusters=east
kubectl --context=${CTX_CLUSTER2} label namespace istio-system topology.istio.io/network=network2

Install Istio service with remote profile, to set up this remote cluster to be managed by a control plane in the primary east cluster:

DISCOVERY_ADDRESS=$(kubectl --context=east -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/west/net/network2
    global:
      remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF

istioctl install -y --context="${CTX_CLUSTER2}" -f cluster2.yaml

Install Prometheus addon:

kubectl --context=${CTX_CLUSTER2} apply -f ${ISTIO_DIR}/samples/addons/prometheus.yaml -n istio-system

Generate gateway in cluster west:

istioctl x create-remote-secret \
    --context="${CTX_CLUSTER2}" \
    --name=west | \
    kubectl apply -f - --context="${CTX_CLUSTER1}"

${ISTIO_DIR}/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster ${CTX_CLUSTER2} --network network2 | ${ISTIO_DIR}/bin/istioctl --context=${CTX_CLUSTER2} install -y -f -

Set up Prometheus federation:

kubectl patch svc prometheus -n istio-system --context ${CTX_CLUSTER2} -p "{\"spec\": {\"type\": \"LoadBalancer\"}}"

WEST_PROMETHEUS_ADDRESS=$(kubectl --context=west -n istio-system get svc prometheus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -L -o p8s.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml

sed -i "s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g" p8s.yaml

kubectl --context="${CTX_CLUSTER1}" apply -f p8s.yaml -n istio-system

There will be a separate article to show how to setup unified Traces.

Install error rates application demo distributed between the two clusters:

curl -L -o install-error-rates-demo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-error-rates-demo.sh
chmod +x install-error-rates-demo.sh
./install-error-rates-demo.sh -c kubectl -dd true

Get IP for Kiali access:

minikube ip -p ${CTX_CLUSTER1}

Kiali multi-cluster demo

At this point, we should be able to access the Kiali instance and see both clusters in the Mesh tab:

We should be able to see graphs from the beta namespace in the remote west cluster:

A video with a demo for an overview of Kiali Multi-cluster features is available:

This is just the first step for Kiali multi-cluster, to help going forward it’s important to get your community user feedback! We’d love to hear from you in comments, in Slack or in Github. Stay tuned for more blog posts, demos and tutorials about Kiali Multi-cluster.