Cross Cluster Service Mesh Using Istio

Implementing service discovery across multiple Kubernetes clusters

Introduction

In this article, I will talk about how you can setup a Service mesh using Istio which spans across multiple clusters. Although the documentation is available from Istio as well, but it gets very confusing and I had a hard time to get everything up and running. I hope this article will help others who are trying to do the same setup and are stuck because of poor documentation.

There are two ways to setup multicluster Istio mesh:

1. Shared Control Plane
2. Replicated Control Plane

You can find more details in the official docs.

I would suggest to go with Replicated control plane setup because it offers better HA, each cluster can be managed independently by different teams and any config issue in one cluster’s control plane will not affect other cluster services. So, I will be explaining about Replicated control plane setup only.

Things to Note

1. In cross cluster service mesh, communication between services require mutual TLS. This means you need a common set of certs to be created for both clusters, this includes having a common root CA cert and intermediate certs for each cluster signed using these root CA certs.
2. Even when you setup multi cluster service mesh, you cannot dynamically discover all services from one cluster to other cluster. You need to add configs for each service you want to talk to from one cluster to another(which I’ll be showing in this article).
3. Cross cluster service calls happen via the istio ingress gateway, so make sure you have a dns of second cluster for calls from first cluster.
4. I did the whole setup in Istio 1.6.4 and would suggest to use Istio version 1.6+ only because it also claims to have lower latency in traffic flow as compared to older versions.
5. The process as per official docs is mentioned here, but I would try making the process more simple and straightforward.

Setup Process

I assume you already have two Kubernetes clusters with Istio installed in them.

To setup replicated control plane cross cluster service mesh you need to do following steps:

1. Create certs for both clusters

This was a tricky part and this is where I was stuck. After lots of searching, found a good article which explains how to create certs easily using cloudflare’s utility “cfssl”. Follow this article to create your root CA certs and intermediate certs. Make sure you set certificate expiry interval according to your needs. You can either create different intermediate certs for both clusters or use the same one.

If you have created the required input files correctly, I will summarize the commands:

$ cfssl gencert -initca ca.json | cfssljson -bare ca

$ cfssl gencert -initca intermediate-ca.json | cfssljson -bare intermediate_ca

$ cfssl sign -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile intermediate_ca intermediate_ca.csr | cfssljson -bare intermediate_ca

With the first command you are creating root certs(At the end you can rename the cert as root-cert.pem), with second command, you’ll create intermediate certs and with third command, you are signing intermediate certs with root certs(At the end you can rename the intermediate cert files as: ca-cert.pem and ca-key.pem).

After this, you’ll have following files:

• root-cert.pem
• ca-cert.pem
• ca-key.pem

Now, create a cert-chain.pem file using the commands:

$ cp ca-cert.pem cert-chain.pem

$ cat root-cert.pem >> cert-chain.pem

If you want you can use any other approach as well to create these certs. To verify that your certs are correct, you can use following commands:

$ openssl verify -CAfile cert-chain.pem ca-cert.pem

$ openssl verify -CAfile root-cert.pem ca-cert.pem

Both the commands should give you the output like:

ca-cert.pem: OK

Now, apply these certs in the clusters using the command:

kubectl create secret generic cacerts -n istio-system \
--from-file=ca-cert.pem \
--from-file=ca-key.pem \
--from-file=root-cert.pem \
--from-file=cert-chain.pem

If you have not yet setup istio, install Istio after this step, if Istio is already installed, recreate istiod and ingress gateway pod.

2. DNS setup for cross cluster calls

So, generally Istio official docs suggest that services of other cluster should be accessed in format: “<service_name>.<namespace>.global”. This can be achieved by adding Istio’s coredns service ip in Kubernetes cluster’s Coredns configmap.

This is how your cluster’s Coredns configmap will look like(replace kubectl part with Istio’s Coredns Service IP):

3. Create services

Now, just create your required services in cluster 1 and cluster 2. Lets say you have deployed service svc-1 in namespace foo in cluster 1 and svc-2 in namespace bar in cluster 2. This is how your communication between services will look like:

Istio cross-cluster traffic flow

4. Make cluster 1 aware of service in cluster 2

To enable a service in cluster 1 to talk to a service in cluster 2, you need to create a Service Entry with following settings:

• Using ingress gateway dns of cluster-2 in the Service Entry file to specify the route to choose for sending traffic to that service.
• A unique IP within the cluster, so that Istio can track all traffic for this IP. This IP need not be resolvable and is just required by Istio for tracking traffic purposes.
• These unique Ips will be used for all *.global domains created in your cluster. Istio suggests to use 240.0.0.0/4 cidr for all these unique IPs

Here is what the ServiceEntry file will look like:

Now, service 1 in cluster 1 can directly call service 2 using dns: “svc-2.bar.global”

Ending Note

Thanks for reading. I hope this article helps you in setting up multi cluster service mesh. Cheers!