Connecting multiple Kubernetes Clusters on vSphere with Istio Service Mesh
In my previous post I had looked at connecting two vSphere clusters using the Cilium Cluster Mesh. Here we try and do the same experiment using Istio
We wont go into the details and capabilities of what Istio is and what it can do as there is numerous material out there . The most essential thing to remember as we later compare this solution with Cilium Cluster Mesh solution is that Istio is a Service mesh and as defined in the istio docs -
The term service mesh is used to describe the network of microservices that make up such applications and the interactions between them
So here we are extending and utilizing the concept of a service mesh to provide a cluster mesh for applications across multiple Kubernetes clusters . This might be an important thing to remember when comparing solutions at the end of this post.
The istio setup and configuration was quite straightforward but setting up cockroachdb across the clusters required quite a bit of tweaking.
Deployment Architecture
The architecture is somewhat similar to my previous experiment -
- Setup multiple independent K8S clusters
- Install Istio with replicated control planes in both the clusters.
- Install a high availability application across the clusters.
For installing the Kubernetes Clusters, I followed the similar steps as in my previous experiment using the Cluster API . I won’t duplicate the steps as they are documented here .
Istio has multiple options for multi cluster installation —
Of the three options above, the single network option requires network connectivity between pods/clusters and the pod CIDRs should not overlap . So we need a solution like cilium. Therefore decided against choosing the option as we already experimented such a setup without istio and just cilium.Between the shared control plane -multi-network and replicated control planes I chose the Replicated Control Planes setup as that provides redundancy for istio services too.
For the high availability application deployment, I chose cockroachdb just like in previous blog post.
Setup
Kubernetes Installation
For installing the Kubernetes Clusters, I followed the similar steps as in my previous experiment using the Cluster API . I won’t duplicate the steps as they are documented here . The only difference being that the pod CIDRs need not be different across the clusters ;they can overlap across the two clusters. We also install metallb as our load balancing solution for vSphere as described previously here .
Istio Replicated Control Plane Installation
In this setup of Istio, the Istio control plane components are installed on both clusters independently with a common root CA to tie things between them. With the common root CA , cross cluster communication occurs over TLS between istio sidecars/egress gateways and ingress gateways. Internal istio local certificate management by citadel requires the intermediate certs generated during root CA generation.
The istio gateways are configured match on a non existent/virtual domain-by default “*.global” . Any requests to a service ending with ‘.global’ domain is sent from the CoreDNS of the native Kubernetes cluster to istio’s coredns which runs a special plugin called istio-coredns-plugin . This plugin serves DNS records by reading Service Entry definitions in istio thereby extending the DNS resolution for services not present in the Kubernetes service registry .
- Before we begin the istio installation we need to generate a root CA , common to both clusters and also generate intermediate CA certificates for use by Citadel for in cluster use. We can use the standard openssl commands for this . Alternatively, we can use the Makefile provided by istio for ease :
>curl -sO https://raw.githubusercontent.com/istio/istio/release-1.4/samples/certs/Makefile>mkdir east-1
>mkdir west-1## Make the root CA> make root-ca
generating root-key.pem
Generating RSA private key, 4096 bit long modulus
.......................................................................................................................................................++
.............................++
e is 65537 (0x10001)
generating root-cert.csr
generating root-cert.pem
Signature ok
subject=/O=Istio/CN=Root CA
Getting Private key# Generate the intermediate CA for both our east and west clusters
> make east-1-certs
generating east-1/ca-key.pem
Generating RSA private key, 4096 bit long modulus
...........................................................................................................................................................................++
...............................................++
e is 65537 (0x10001)
generating east-1/cluster-ca.csr
generating east-1/ca-cert.pem
Signature ok
subject=/O=Istio/CN=Intermediate CA/L=east-1
Getting CA Private Key
generating east-1/cert-chain.pem
Citadel inputs stored in east-1/> make west-1-certs
generating west-1/ca-key.pem
Generating RSA private key, 4096 bit long modulus
.......................................................................................................++
........................................................++
e is 65537 (0x10001)
generating west-1/cluster-ca.csr
generating west-1/ca-cert.pem
Signature ok
subject=/O=Istio/CN=Intermediate CA/L=west-1
Getting CA Private Key
generating west-1/cert-chain.pem
Citadel inputs stored in west-1/
2. Next let us create the istio namespaces and the secrets
>kubectl create namespace istio-system --context=k8s-cluster-east-1
namespace/istio-system created
>kubectl create namespace istio-system --context=k8s-cluster-west-1
namespace/istio-system created## create the secrets
> kubectl create secret generic cacerts -n istio-system \
--from-file=west-1/ca-cert.pem \
--from-file=west-1/ca-key.pem \
--from-file=west-1/root-cert.pem \
--from-file=west-1/cert-chain.pem --context=k8s-cluster-west-1
secret/cacerts created> kubectl create secret generic cacerts -n istio-system \
--from-file=east-1/ca-cert.pem \
--from-file=east-1/ca-key.pem \
--from-file=east-1/root-cert.pem \
--from-file=east-1/cert-chain.pem --context=k8s-cluster-east-1
secret/cacerts created
3. Download the istio release ,set istioctl client tool in the path .
> curl -sL https://istio.io/downloadIstio | sh -
Downloading istio-1.4.2 from https://github.com/istio/istio/releases/download/1.4.2/istio-1.4.2-osx.tar.gz ... % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 612 0 612 0 0 2072 0 --:--:-- --:--:-- --:--:-- 2074
100 32.4M 100 32.4M 0 0 2426k 0 0:00:13 0:00:13 --:--:-- 1981k
Istio 1.4.2 Download Complete!Istio has been successfully downloaded into the istio-1.4.2 folder on your system.Next Steps:
See https://istio.io/docs/setup/kubernetes/install/ to add Istio to your Kubernetes cluster.To configure the istioctl client tool for your workstation,
add the /istio-1.4.2/bin directory to your environment path variable with:
export PATH="$PATH:/istio-1.4.2/bin"Begin the Istio pre-installation verification check by running:
istioctl verify-installNeed more information? Visit https://istio.io/docs/setup/kubernetes/install/
4. Install istio
# Create the namespace
> kubectl create namespace istio-system --context=k8s-cluster-east-1
namespace/istio-system created> kubectl create namespace istio-system --context=k8s-cluster-west-1
namespace/istio-system created# Apply the istio manifest from the istio install directory to both the clustersistio-1.4.2 > istioctl manifest apply -f ./install/kubernetes/operator/examples/multicluster/values-istio-multicluster-gateways.yaml --set values.global.proxy.accessLogFile="/dev/stdout" --context=k8s-cluster-east-1Preparing manifests for these components:
- PrometheusOperator
- Base
- Injector
- Tracing
- Prometheus
- Telemetry
- Pilot
- Kiali
- CoreDNS
- NodeAgent
- EgressGateway
- IngressGateway
- CertManager
- Grafana
- Policy
- Citadel
- Cni
- GalleyApplying manifest for component CoreDNS
Applying manifest for component Base
Finished applying manifest for component CoreDNS
Finished applying manifest for component Base
Applying manifest for component Citadel
Applying manifest for component Policy
Applying manifest for component IngressGateway
Applying manifest for component EgressGateway
Applying manifest for component Galley
Applying manifest for component Prometheus
Applying manifest for component Pilot
Applying manifest for component Injector
Applying manifest for component Telemetry
Finished applying manifest for component Citadel
Finished applying manifest for component Prometheus
Finished applying manifest for component Galley
Finished applying manifest for component Injector
Finished applying manifest for component IngressGateway
Finished applying manifest for component Policy
Finished applying manifest for component Pilot
Finished applying manifest for component EgressGateway
Finished applying manifest for component TelemetryComponent Kiali installed successfully:
=======================================
Component CoreDNS installed successfully:
=========================================
Component NodeAgent installed successfully:
===========================================
Component EgressGateway installed successfully:
===============================================
Component IngressGateway installed successfully:
================================================
Component CertManager installed successfully:
=============================================
Component Grafana installed successfully:
=========================================
Component Policy installed successfully:
========================================
Component Citadel installed successfully:
=========================================
Component Cni installed successfully:
=====================================
Component Galley installed successfully:
========================================
Component PrometheusOperator installed successfully:
====================================================
Component Base installed successfully:
======================================
Component Injector installed successfully:
==========================================
Component Tracing installed successfully:
=========================================
Component Prometheus installed successfully:
============================================
Component Telemetry installed successfully:
===========================================
Component Pilot installed successfully:
=======================================istio-1.4.2 > istioctl manifest apply -f ./install/kubernetes/operator/examples/multicluster/values-istio-multicluster-gateways.yaml --set values.global.proxy.accessLogFile="/dev/stdout" --context=k8s-cluster-west-1Preparing manifests for these components:
- PrometheusOperator
- Base
- Injector
- Tracing
- Prometheus
- Telemetry
- Pilot
- Kiali
- CoreDNS
- NodeAgent
- EgressGateway
- IngressGateway
- CertManager
- Grafana
- Policy
- Citadel
- Cni
- GalleyApplying manifest for component CoreDNS
Applying manifest for component Base
Finished applying manifest for component CoreDNS
Finished applying manifest for component Base
Applying manifest for component Citadel
Applying manifest for component Policy
Applying manifest for component IngressGateway
Applying manifest for component EgressGateway
Applying manifest for component Galley
Applying manifest for component Prometheus
Applying manifest for component Pilot
Applying manifest for component Injector
Applying manifest for component Telemetry
Finished applying manifest for component Citadel
Finished applying manifest for component Prometheus
Finished applying manifest for component Galley
Finished applying manifest for component Injector
Finished applying manifest for component IngressGateway
Finished applying manifest for component Policy
Finished applying manifest for component Pilot
Finished applying manifest for component EgressGateway
Finished applying manifest for component TelemetryComponent Kiali installed successfully:
=======================================
Component CoreDNS installed successfully:
=========================================
Component NodeAgent installed successfully:
===========================================
Component EgressGateway installed successfully:
===============================================
Component IngressGateway installed successfully:
================================================
Component CertManager installed successfully:
=============================================
Component Grafana installed successfully:
=========================================
Component Policy installed successfully:
========================================
Component Citadel installed successfully:
=========================================
Component Cni installed successfully:
=====================================
Component Galley installed successfully:
========================================
Component PrometheusOperator installed successfully:
====================================================
Component Base installed successfully:
======================================
Component Injector installed successfully:
==========================================
Component Tracing installed successfully:
=========================================
Component Prometheus installed successfully:
============================================
Component Telemetry installed successfully:
===========================================
Component Pilot installed successfully:
5. As mentioned above the ingress gateways on each cluster responds to requests for services on a ‘.global’ domain in this setup of istio. To forward requests from one cluster to another for these services, each local coredns needs to forward such requests to istio’s own coredns which runs with a plugin designed to read service information from istio’s service entry. To achieve this we first get the cluster IPs of coredns installed by istio
>kubectl get svc -n istio-system istiocoredns -o jsonpath={.spec.clusterIP} --context=k8s-cluster-east-1
101.66.100.253
>kubectl get svc -n istio-system istiocoredns -o jsonpath={.spec.clusterIP} --context=k8s-cluster-west-1
101.66.159.57Then we edit the coredns config map in each cluster to forward all such service requests to the respective istiocoredns cluster ips from above . Our configmaps look like this
5. It is time to test the cluster connectivity. We use the sample ‘sleep’ and ‘httpbin’ deployments from the istio install directory distributing them between the clusters.
# First create our namespaces in both the clusters
> kubectl create namespace east-1 --context=k8s-cluster-east-1
namespace/east-1 created> kubectl create namespace west-1 --context=k8s-cluster-west-1
namespace/west-1 created# Enable istio injection on each namespace
> kubectl label namespace east-1 istio-injection=enabled --context=k8s-cluster-east-1
namespace/east-1 labeled
>kubectl label namespace west-1 istio-injection=enabled --context=k8s-cluster-west-1
namespace/west-1 labeled# Deploy the sample Sleep service on the east-1 namespace and cluster and the httpbin service on the west-1 namespace and cluster>kubectl apply -f samples/sleep/sleep.yaml -n east-1 --context=k8s-cluster-east-1
serviceaccount/sleep created
service/sleep created
deployment.apps/sleep created> kubectl apply -f samples/httpbin/httpbin.yaml -n west-1 --context=k8s-cluster-west-1
serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created
6. Now , for the sleep service in the east-1 cluster to connect to the httpbin service on west-1 cluster , we need a way of routing this request to the ingress gateway of the west-1 cluster from the istio sidecar in the east-1 cluster. The istio installation for multiple control planes by default uses a ‘.global’ domain to mark such services which are remotely available and we use the istio ServiceEntry to provide the information of the remote service. The service entry essentially extends the service registry of kubernetes for provide custom service definitions which istio can process.
# First we find the ingress gateway LB ip of the west-1 cluster> kubectl get svc --selector=app=istio-ingressgateway -n istio-system -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}' --context=k8s-cluster-west-1
10.11.84.71## Then we create a ServiceEntry for the httpbin.west-1.global service in the east-1 cluster as below
Now we test the connectivity
> kubectl get pods -n east-1 --context=k8s-cluster-east-1
NAME READY STATUS RESTARTS AGE
sleep-6bdb595bcb-rb89p 2/2 Running 0 6h18m>kubectl exec -it sleep-6bdb595bcb-rb89p -c sleep -n east-1 --context=k8s-cluster-east-1 -- curl httpbin.west-1.global:8000/headers{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "httpbin.west-1.global:8000",
"User-Agent": "curl/7.64.0",
"X-B3-Parentspanid": "5d7aca6945aafd60",
"X-B3-Sampled": "0",
"X-B3-Spanid": "9a4f59e77f8b0451",
"X-B3-Traceid": "0a7ed2114a2e67055d7aca6945aafd60",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/west-1/sa/httpbin;Hash=6fbc06731c682bf1f68d980fe0397b11962115d4742c1564925e000b7be4e560;Subject=\"\";URI=spiffe://cluster.local/ns/east-1/sa/sleep"
}
}
As we can see above, the two services can connect and respond to each other across the clusters. Thats about it for the istio setup. Now we can look at deploying Cockroachdb across the two clusters.
Setup CockroachDB Across Multiple Kubernetes Clusters
As we did my previous post , we will follow the official cockroachdb documentation which outlines these steps for doing on GKE here but with istio there are some differences in the concept as compared to when we used cilium and clusters having inter pod connectivity -
- Unlike the cockroachdb documentation steps ,we don’t need to expose the coredns endpoints with a public load balancers when using istio. The istio ingress gateway already does this.
- While the above test using the ‘sleep’ and ‘httpbin’ worked seamlessly and quite easily, the same concept cant be applied verbatim to StatefulSets which the cockroachdb uses .This is because ,fundamentally istio being a Service Mesh works with services with multiple or alternate endpoints. With StatefulSets creating headless services this doesn’t work. The relevant explanation is here. So in order for istio to understand each headless service endpoint, they all need to declared in istio’s service registry individually. This causes a potential limitation with scaling or auto scaling because each endpoint then has to be added to the ServiceEntry .
- Istio has certain naming conventions when naming ports as described here. The relevant line
Named service ports: Service ports must be named. The port name key/value pairs must have the following syntax:
name: <protocol>[-<suffix>].
The yaml files for the StatefulSets need to be tweaked to follow the naming convention and use ‘tcp’ instead of ‘grpc’ as that conflicts with istio’s naming .The relevant issues are discussed in detail here and here. The final flow of traffic will look something like illustrated below.
With the above things in mind , let us make the relevant adjustments and deploy Cockroachdb .
First download all the multi region configuration files for cockroach
>curl -OOOOOOOOO https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/{README.md,client-secure.yaml,cluster-init-secure.yaml,cockroachdb-statefulset-secure.yaml,dns-lb.yaml,example-app-secure.yaml,external-name-svc.yaml,setup.py,teardown.py}Create the cockroach ca and client certificates and the appropriate cockroach secrets containing the node and client certificates in all the clusters. The client certificate is created in both the region/zone namespace and default namespace for ease of access to the services.
> mkdir certs
> mkdir ca-keys
> cockroach cert create-ca --certs-dir certs --ca-key ca-keys/ca.key
> cockroach cert create-client root --certs-dir certs --ca-key ca-keys/ca.key> kubectl create secret generic cockroachdb.client.root --from-file certs --context=k8s-cluster-east-1
> kubectl create secret generic cockroachdb.client.root --from-file certs --context=k8s-cluster-west-1> kubectl create secret generic cockroachdb.client.root --from-file certs -n west-1 --context=k8s-cluster-west-1
> kubectl create secret generic cockroachdb.client.root --from-file certs -n east-1 --context=k8s-cluster-east-1#### Create the node certs and secrets for each region/zone> cockroach cert create-node --certs-dir certs --ca-key ca-keys/ca.key localhost 127.0.0.1 cockroachdb-public cockroachdb-public.default cockroachdb-public.west-1 cockroachdb-public.west-1.svc.cluster.local *.cockroachdb *.cockroachdb.west-1 *.cockroachdb.west-1.svc.cluster.local> kubectl create secret generic cockroachdb.node -n west-1 --from-file certs --context=k8s-cluster-west-1> rm -rf certs/node.*> cockroach cert create-node --certs-dir certs --ca-key ca-keys/ca.key localhost 127.0.0.1 cockroachdb-public cockroachdb-public.default cockroachdb-public.east-1 cockroachdb-public.east-1.svc.cluster.local *.cockroachdb *.cockroachdb.east-1 *.cockroachdb.east-1.svc.cluster.local> kubectl create secret generic cockroachdb.node -n east-1 --from-file certs --context=k8s-cluster-east-1
We create the two external services in the default namespaces in each zone .
> sed 's/YOUR_ZONE_HERE/east-1/g' external-name-svc.yaml > external-name-svc-east-1.yaml> kubectl apply -f external-name-svc-east-1.yaml --context=k8s-cluster-east-1
service/cockroachdb-public created
serviceaccount/cockroachdb created
role.rbac.authorization.k8s.io/cockroachdb created
rolebinding.rbac.authorization.k8s.io/cockroachdb created> sed 's/YOUR_ZONE_HERE/west-1/g' external-name-svc.yaml > external-name-svc-west-1.yaml> kubectl apply -f external-name-svc-west-1.yaml --context=k8s-cluster-west-1
service/cockroachdb-public created
serviceaccount/cockroachdb created
role.rbac.authorization.k8s.io/cockroachdb created
rolebinding.rbac.authorization.k8s.io/cockroachdb created## Verify>kubectl get svc cockroachdb-public --context=k8s-cluster-east-1NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cockroachdb-public ExternalName <none> cockroachdb-public.east-1.svc.cluster.local <none> 2m19s>kubectl get svc cockroachdb-public --context=k8s-cluster-west-1NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cockroachdb-public ExternalName <none> cockroachdb-public.west-1.svc.cluster.local <none> 2m4s
Before we tweak our StatefulSet yaml files for istio and deploy, we shall create the relevant istio specific resources definitions to direct traffic from one cluster to another .
First we create the ServiceEntry configurations for each cockroachdb node in a cluster. Since our StatefulSet is a 3 node cluster we have 3 of these . As you observe we have the relevant ports for cockroachdb i.e 26257 and 8080 targeted here for each headless service.
> kubectl apply -f sventry-crdb-east-1.yaml --context=k8s-cluster-east-1serviceentry.networking.istio.io/cockroachdb-0-headless-service-entry created
serviceentry.networking.istio.io/cockroachdb-1-headless-service-entry created
serviceentry.networking.istio.io/cockroachdb-2-headless-service-entry created>kubectl apply -f sventry-crdb-west-1.yaml --context=k8s-cluster-west-1
serviceentry.networking.istio.io/cockroachdb-0-headless-service-entry created
serviceentry.networking.istio.io/cockroachdb-1-headless-service-entry created
serviceentry.networking.istio.io/cockroachdb-2-headless-service-entry created## Verify
>kubectl get se -n east-1 --context=k8s-cluster-east-1
NAME HOSTS LOCATION RESOLUTION AGE
cockroachdb-0-headless-service-entry [cockroachdb-0.cockroachdb.east-1.svc.cluster.local] MESH_INTERNAL DNS 3m2s
cockroachdb-1-headless-service-entry [cockroachdb-1.cockroachdb.east-1.svc.cluster.local] MESH_INTERNAL DNS 3m2s
cockroachdb-2-headless-service-entry [cockroachdb-2.cockroachdb.east-1.svc.cluster.local] MESH_INTERNAL DNS 3m2s> kubectl get se -n west-1 --context=k8s-cluster-west-1
NAME HOSTS LOCATION RESOLUTION AGE
cockroachdb-0-headless-service-entry [cockroachdb-0.cockroachdb.west-1.svc.cluster.local] MESH_INTERNAL DNS 20s
cockroachdb-1-headless-service-entry [cockroachdb-1.cockroachdb.west-1.svc.cluster.local] MESH_INTERNAL DNS 20s
cockroachdb-2-headless-service-entry [cockroachdb-2.cockroachdb.west-1.svc.cluster.local] MESH_INTERNAL DNS 20s
Next we need to create Service Entries for the remote .global service in each cluster i.e. e.g east-1 cluster needs to have service entries for cockroachdb-0.cockroachdb.west-1.global and vice versa. The configurations are as below
> kubectl apply -f cockroachdb-west-1-global-serviceentry.yaml -n east-1 --context=k8s-cluster-east-1 serviceentry.networking.istio.io/cockroachdb0-west-1-global created
serviceentry.networking.istio.io/cockroachdb1-west-1-global created
serviceentry.networking.istio.io/cockroachdb2-west-1-global created
serviceentry.networking.istio.io/cockroachdb-public-west-1-global created> kubectl apply -f cockroachdb-east-1-global-serviceentry.yaml -n west-1 --context=k8s-cluster-west-1serviceentry.networking.istio.io/cockroachdb0-east-1-global created
serviceentry.networking.istio.io/cockroachdb1-east-1-global created
serviceentry.networking.istio.io/cockroachdb2-east-1-global created
serviceentry.networking.istio.io/cockroachdb-public-east-1-global created
We created Service Entries for cockroachdb headless services so they can addressed like cockroachdb-0.cockroachdb.west-1.global ,cockroachdb-0.cockroachdb.east-1.global etc . Typically this is enough for connecting services across clusters using the istio ingress gateway.
But our cockroachdb cluster itself knows no difference between a local and a global node . So when we provide a list of nodes for a cockroachdb cluster to join, we want our nodes to be named with the same convention on both clusters . In order to achieve this, we should be able to address the remote services without the “.global” suffix . For this we add another set of service entries without that suffix.
> kubectl apply -f cockroachdb-west-1-noglobal-serviceentry.yaml --context=k8s-cluster-east-1serviceentry.networking.istio.io/cockroachdb0-west-1 created
serviceentry.networking.istio.io/cockroachdb1-west-1 created
serviceentry.networking.istio.io/cockroachdb2-west-1 created
serviceentry.networking.istio.io/cockroachdb-public-west-1 created> kubectl apply -f cockroachdb-east-1-noglobal-serviceentry.yaml --context=k8s-cluster-west-1
serviceentry.networking.istio.io/cockroachdb0-east-1 created
serviceentry.networking.istio.io/cockroachdb1-east-1 created
serviceentry.networking.istio.io/cockroachdb2-east-1 created
serviceentry.networking.istio.io/cockroachdb-public-east-1 created
But that is not enough. Istio uses the host header/SNI header to route traffic to services appropriately so we need to also add a Virtual Service . The Virtual Service adds a host header to each service addressed without the “.global” service.
>kubectl apply -f cockroachdb-west-1-vs.yaml --context=k8s-cluster-east-1
virtualservice.networking.istio.io/cockroachdb-0.cockroachdb.west-1 created
virtualservice.networking.istio.io/cockroachdb-1.cockroachdb.west-1 created
virtualservice.networking.istio.io/cockroachdb-2.cockroachdb.west-1 created
virtualservice.networking.istio.io/cockroachdb-public.west-1 created>kubectl apply -f cockroachdb-east-1-vs.yaml --context=k8s-cluster-west-1
virtualservice.networking.istio.io/cockroachdb-0.cockroachdb.east-1 created
virtualservice.networking.istio.io/cockroachdb-1.cockroachdb.east-1 created
virtualservice.networking.istio.io/cockroachdb-2.cockroachdb.east-1 created
virtualservice.networking.istio.io/cockroachdb-public.east-1 created
We are finally ready to deploy our StatefulSets. First we generate our join list string of all the cockroachdb instances/pods and apply to the provided template file .
>JOINSTR="cockroachdb-0.cockroachdb.east-1,cockroachdb-1.cockroachdb.east-1,cockroachdb-2.cockroachdb.east-1,cockroachdb-0.cockroachdb.west-1,cockroachdb-1.cockroachdb.west-1,cockroachdb-2.cockroachdb.west-1"> sed 's/JOINLIST/'"${JOINSTR}"'/g;s/LOCALITYLIST/zone=east-1/g' cockroachdb-statefulset-secure.yaml > cockroachdb-statefulset-secure-east-1.yaml> sed 's/JOINLIST/'"${JOINSTR}"'/g;s/LOCALITYLIST/zone=west-1/g' cockroachdb-statefulset-secure.yaml > cockroachdb-statefulset-secure-west-1.yaml
Next , edit each of the StatefulSet yaml files to change the names of the grpc ports in the Service definitions
....
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
name: cockroachdb-public
labels:
app: cockroachdb
spec:
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: tcp-crdbpublic1
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: crdbpublic2
selector:
app: cockroachdb
---
apiVersion: v1
kind: Service
metadata:
# This service only exists to create DNS entries for each pod in the stateful
# set such that they can resolve each other's IP addresses. It does not
# create a load-balanced ClusterIP and should not be used directly by clients
# in most circumstances.
name: cockroachdb
labels:
app: cockroachdb
annotations:
# Use this annotation in addition to the actual publishNotReadyAddresses
# field below because the annotation will stop being respected soon but the
# field is broken in some versions of Kubernetes:
# https://github.com/kubernetes/kubernetes/issues/58662
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
# Enable automatic monitoring of all instances when Prometheus is running in the cluster.
prometheus.io/scrape: "true"
prometheus.io/path: "_status/vars"
prometheus.io/port: "8080"
spec:
ports:
- port: 26257
targetPort: 26257
name: tcp-crdbheadless1
- port: 8080
targetPort: 8080
name: crdbheadless2
# We want all pods in the StatefulSet to have their addresses published for
# the sake of the other CockroachDB pods even before they're ready, since they
# have to be able to talk to each other in order to become ready.
publishNotReadyAddresses: true
clusterIP: None
selector:
app: cockroachdbThere is one last change we need to make . We notice our join string above does not contain the FQDN of each node e.g. cockroachdb-0.cockroachdb.east-1 . But when a node from the west-1 cluster e.g. cockroachdb-0.cockroachdb.west-1 connects to cockroachdb-0.cockroachdb.east-1 , the east-1 node responds with information about itself with a fully qualified domain address i.e. cockroachdb-0.cockroachdb.east-1.svc.cluster.local. Subsequently the west-1 node tries to reach this address which obviously fails . To fix this we need to change the “ — advertise-host” setting in the cockroach start command . By default it is
>exec /cockroach/cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host $(hostname -f) ....we change this to
exec /cockroach/cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host $(hostname -f |sed 's/.svc.cluster.local//g') Once we do this, thanks to our VirtualService and ServiceEntry , nodes can reach each other without needing to resolve the FQDN of each other. Our final config files look like this
We also disable istio sidecar injection for the cluster init job.
Time to apply all these configurations.
>kubectl apply -f cockroachdb-statefulset-secure-east-1.yaml -n east-1 --context=k8s-cluster-east-1
serviceaccount/cockroachdb created
role.rbac.authorization.k8s.io/cockroachdb created
clusterrole.rbac.authorization.k8s.io/cockroachdb created
rolebinding.rbac.authorization.k8s.io/cockroachdb created
clusterrolebinding.rbac.authorization.k8s.io/cockroachdb created
service/cockroachdb-public created
service/cockroachdb created
poddisruptionbudget.policy/cockroachdb-budget created
statefulset.apps/cockroachdb created## Check the pods are running
>kubectl get pods -n east-1 --context=k8s-cluster-east-1
NAME READY STATUS RESTARTS AGE
cockroachdb-0 1/2 Running 0 19s
cockroachdb-1 1/2 Running 0 19s
cockroachdb-2 1/2 Running 0 19s#initialize the cluster
> kubectl apply -f cluster-init-secure.yaml -n east-1 --context=k8s-cluster-east-1
job.batch/cluster-init-secure created## All containers go to ready state
>kubectl get pods -n east-1 --context=k8s-cluster-east-1
NAME READY STATUS RESTARTS AGE
cluster-init-secure-cpr4c 0/1 Completed 0 10s
cockroachdb-0 2/2 Running 0 63s
cockroachdb-1 2/2 Running 0 63s
cockroachdb-2 2/2 Running 0 63s## the configuration in the west-1 cluster
>kubectl apply -f cockroachdb-statefulset-secure-west-1.yaml -n west-1 --context=k8s-cluster-west-1serviceaccount/cockroachdb created
role.rbac.authorization.k8s.io/cockroachdb created
clusterrole.rbac.authorization.k8s.io/cockroachdb created
rolebinding.rbac.authorization.k8s.io/cockroachdb created
clusterrolebinding.rbac.authorization.k8s.io/cockroachdb created
service/cockroachdb-public created
service/cockroachdb created
poddisruptionbudget.policy/cockroachdb-budget created
statefulset.apps/cockroachdb created## Verify all containers go to running state
>kubectl get pods -n west-1 --context=k8s-cluster-west-1
NAME READY STATUS RESTARTS AGE
cockroachdb-0 2/2 Running 0 41s
cockroachdb-1 2/2 Running 0 41s
cockroachdb-2 2/2 Running 0 41s
Now verify the cluster by creating some test data as described here Test Cluster Setup
We can also check via the cockroachdb web console that pods across the kubernetes clusters form a unified cockroachdb cluster. As we can see the nodes all have the domain name ending with the namespace with “svc.cluster.local” omitted due to our “ — advertise-host” setting.
Conclusions and comparisons with the Cilium Cluster Mesh
Just like the solution using Cilium in my previous post, we saw how one can achieve the same result of clustering applications across multiple kubernetes clusters.
The road to the solutions ,though are quite different.
The Cilium Cluster Mesh as the name suggests is well a Cluster Mesh operating at the CNI level but requires networking connectivity between the nodes across the clusters and non overlapping pod IP CIDRs and typically needs to be setup when deploying a Kubernetes cluster. But once setup it behaves seamlessly like a single cluster with the routable pod IPs so the applications don’t have to do anything extra a multi cluster distribution. TLS is not builtin to communication between the clusters though and that needs to be handled by the application .
The Istio Service Mesh operates at the service level. It does not care for overlapping pod IP CIDRs and uses ingress gateways with mTLS between the two clusters to achieve communication.Naturally operating at this level requires additional istio specific resource definitions for application deployments which falls in the lap of the application developer. Some of it probably can be automated using custom controllers or kubernetes webhooks. If a project application doesn’t use istio at all then it can really be an over head deploying all the istio components .Also considering all the hops a request goes through from container to sidecar via mTLS parsing all the Service Entry rules and processing requests on the remote gateway can potentially be a performance overhead.
A custom solution with an api gateway is something people might prefer building on their own. (at the risk of duplicating everything istio provides eventually )
Personally , I feel the answer for a preferred solution between two doesn’t exist as they are really two different tools for different needs .If nodes are routable between the clusters and on fresh K8S setups, just using Cilium Cluster Mesh is suffice. Otherwise istio or any custom gateway based will do. If already using Istio as a service mesh in an application then its quite natural to extend it to handle multi cluster setups.
The real interesting topic which I will try and explore in a future post is how Istio can benefit with Cilium , not limited to multi cluster capabilities.
References
- Istio Multi-cluster with Replicated Control Planes
- Deployment of CockroachDB in Istio Injected Namespace
- Making Istio Work with Kubernetes StatefulSet and Headless Services
Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬.
To join our community Slack team chat 🗣️ read our weekly Faun topics 🗞️, and connect with the community 📣 click here⬇
