Istio multicluster with istio-csr + cert-manager + vault PKI

10 min readApr 23, 2024

Goal

Certificate published by Vault PKI, using cert-manager for handling cert from vault, and istio-csr to request cert from cert-manager and distribute cert to all istio-proxy (sidecar) & istiod. and can communicate across cluster kubernetes.

Prerequisite

vault v1.16.1
istio v1.20.0
cert-manager v1.13.2
istio-csr 0.7.1

Architecture

multicluster (multi primary) single network

Introduction

Root and Intermediate Certificate Authority (CA)

The root CA is the top-level certificate authority that signs the intermediate CA certificate. It establishes the trust hierarchy within the PKI.

Vault

Vault can be configured to act as a CA (Certificate Authority) for Istio multicluster deployments. It can manage both the root CA and intermediate CA certificates, providing a secure foundation for issuing certificates within and across clusters.

cert-manager

cert-manager can be configured to issue certificates for Istio components, such as Ingress Gateways, using Vault or Let’s Encrypt as the certificate issuer. This includes defining certificate requests, policies, and renewal settings.

istio-csr

istio-csr can be utilized to handle certificate signing requests for services that need to communicate securely between clusters. This includes generating certificates for Istio gateways and services that span multiple clusters.

Step 1 Deploy Vault

step 1. Add helm repo hashicorp

helm repo add hashicorp https://helm.releases.hashicorp.com

step 2. deploy vault using helm chart

helm upgrade --install --version 0.27.0 vault hashicorp/vault

step 3. init vault to get unseal key. keep this unseal key.

 kubectl exec -it vault-0 -- vault operator init

and you get unseal key 1–5, also root token for login to vault.

Unseal Key 1: szzicGp+99ZIT0bJ5QVriy+ItI319HzYiRpVUg8+Qmq0
Unseal Key 2: gWcDh4cF24D+YRlDDjjtScBF5+jLzdTh3AYdqjYyieJZ
Unseal Key 3: w1LheoTZjMZxUmU4j1SAnZXc+r0hnbziLz022LdT4DTr
Unseal Key 4: 7vz8OSU9eS7Xr32QEgG6s5JiwZSYvTH2vI7LJE5wDiMw
Unseal Key 5: xpbBRO/u0OjuiN+YaywblMcuKlPMWnmMY5mFMWg5v5DG

Initial Root Token: hvs.OjlXv5Yz80QAtpeEJoEKPSpA

step 4. unseal with key above just key 1 to key 3.

kubectl exec -it vault-0 -- vault operator unseal szzicGp+99ZIT0bJ5QVriy+ItI319HzYiRpVUg8+Qmq0
kubectl exec -it vault-0 -- vault operator unseal gWcDh4cF24D+YRlDDjjtScBF5+jLzdTh3AYdqjYyieJZ
kubectl exec -it vault-0 -- vault operator unseal w1LheoTZjMZxUmU4j1SAnZXc+r0hnbziLz022LdT4DTr

Step 2 Create Vault PKI root CA

in this step, we set up the Vault PKI root CA, enabling the pki secrets engine, tuning it for certificate issuance, generating the root CA, and configuring the CA and CRL distribution points.

step 1. install vault-cli and export env VAULT_TOKEN=<YOUR ROOT TOKEN GENERATE FROM VAULT INIT> & VAULT_ADDR=<YOUR VAULT ADDRESS>

note: you can running command `kubectl port-forward svc/vault 8200:8200` and your vault address is 127.0.0.1:8200

export VAULT_TOKEN=<VAULT_TOKEN>
export VAULT_ADDR=<your vault address>

step 2. Enable pki secrets engine to path pki_root

vault secrets enable -path=pki_root pki

step 3. tune pki to issue certificate with Maximum TTL 87600 hours or 10 year.

vault secrets tune -max-lease-ttl=87600h pki_root

step 4. generate root CA with file name CA_cert.crt

common_name is string, you can fill anyting. but i recommend to fill your domain.
ttl is time-to-live your expired certificate. i set ttl same with maximum TTL on point 3.

vault write -field=certificate pki_root/root/generate/internal \
  common_name="yourdomain.com" ttl=87600h > CA_cert.crt

step 5. configure CA and CRL change vault.yourdomain.com:8200 to your url vault.

vault write pki_root/config/urls \
    issuing_certificates="vault.yourdomain.com:8200/v1/pki_root/ca" \
    crl_distribution_points="vault.yourdomain.com:8200/v1/pki_root/crl"

Step 3 Create Vault PKI intermediate CA

This step involves creating and intermediate PKI for the istio cluster, enabling the pki secrets engine, tuning it, generating a CSR for the intermediate CA, signing it with the root CA, creating a certificate chain, and setting the signed intermediate CA.

step 1. Enable pki secrets engine to path pki_int1_istio-cluster1

vault secrets enable -path=pki_int1_istio-cluster1 pki

step 2. tune pki to issue certificate with Maximum TTL 43800 hours or 5 year.

vault secrets tune -max-lease-ttl=43800h pki_int1_istio-cluster1

step 3. generate intermediate CA CSR with file name pki_intermediate1_istio_cluster1.csr

common_name is string, you can fill anyting. but i recommend to fill your domain.
ttl is time-to-live your expired certificate. i set ttl same with maximum TTL on point 3.

vault write -format=json pki_int1_istio-cluster1/intermediate/generate/internal \
        common_name="yourdomain.com intermediate 1" \
        | jq -r '.data.csr' > pki_intermediate1_istio_cluster1.csr

step 4. sign intermediate CA CSR with the root CA and generate file `intermediate1_istio_cluster1.cert.pem`

vault write -format=json pki_root/root/sign-intermediate csr=@pki_intermediate1_istio_cluster1.csr \
        format=pem ttl="43800h" \
        | jq -r '.data.certificate' > intermediate1_istio_cluster1.cert.pem

step 5. create certificate chain to combines the intermediate CA and root CA into a single file.

cat intermediate1_istio_cluster1.cert.pem > intermediate1_istio_cluster1.chain.pem
cat CA_cert.crt >> intermediate1_istio_cluster1.chain.pem

step 6. set signed intermediate CA.

vault write pki_int1_istio-cluster1/intermediate/set-signed certificate=@intermediate1_istio_cluster1.chain.pem

step 7. create role for pki_int1_istio-cluster1 with name istio-ca-istio-cluster1 for specifying various constraints such as allowed_domains, max_ttl,etc.

vault write pki_int1_istio-cluster1/roles/istio-ca-istio-cluster1 \
    allowed_domains=istio-ca \
    allow_any_name=true  \
    enforce_hostnames=false \
    require_cn=false \
    allowed_uri_sans="spiffe://*" \
    max_ttl=72h

step 8. repeat step 1 to step 7 for istio-cluster2. and rename:

pki_int1_istio-cluster1 to pki_int1_istio-cluster2
pki_intermediate1_istio_cluster1.csr to pki_intermediate1_istio_cluster2.csr
intermediate1_istio_cluster1.cert.pem to intermediate1_istio_cluster2.cert.pem
intermediate1_istio_cluster1.chain.pem to intermediate1_istio_cluster2.chain.pem
istio-ca-istio-cluster1 to istio-ca-istio-cluster2

Step 4 Deploy and Configure cert-manager

this step we deploy cert-manager using Helm, deploy CRDs, create a secret for the vault token, and configure cert-manager to communicate with vault for issuing certificates.

step 1. add helm repo and update.

helm repo add jetstack https://charts.jetstack.io
helm repo update

step 2. deploy crds cert-manager.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.crds.yaml

step 3. create namespace cert-manager & istio-system

kubectl create ns cert-manager
kubectl create ns istio-system

step 4. deploy cert-manager.

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.13.2

step 5. create secret for token vault

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cert-manager-vault-token
  namespace: istio-system
data:
  token: <your vault token with base64 encode>
EOF

step 6. add issuer to vault

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault
  namespace: istio-system
spec:
  vault:
    server: http://vault.yourdomain.com:8200
    path: pki_int1_istio-cluster1/sign/istio-ca-istio-cluster1
    auth:
      tokenSecretRef:
          name: cert-manager-vault-token
          key: token
EOF

step 6. repeat step 1 to step 6 for istio-cluster2. at step 5 change path pki_int1_istio-cluster1/sign/istio-ca-istio-cluster1to pki_int1_istio-cluster2/sign/istio-ca-istio-cluster2

Step 5 Deploy istio-csr

this step we pull values.yaml for istio-csr, modify configurations for issuer, certificate DNS names, and istio cluster id, and deploy istio-csr using Helm, ensuring seamless integration with istio components.

step 1. pull values.yaml from https://github.com/cert-manager/istio-csr/blob/main/deploy/charts/istio-csr/values.yaml

step 2. change line 93–102 to:

    issuer:
      # -- Issuer name set on created CertificateRequests for both istio-csr's
      # serving certificate and incoming gRPC CSRs.
      name: vault
      # -- Issuer kind set on created CertificateRequests for both istio-csr's
      # serving certificate and incoming gRPC CSRs.
      kind: Issuer
      # -- Issuer group name set on created CertificateRequests for both
      # istio-csr's serving certificate and incoming gRPC CSRs.
      group: cert-manager.io

step 3. change line 115 to:

    certificateDNSNames:
    - cert-manager-istio-csr.istio-system.svc

step 4. change line 139 to:

  server:
    # -- The istio cluster ID to verify incoming CSRs.
    clusterID: "istio-cluster1"

step 5. deploy istio-csr

helm upgrade --install -n istio-system cert-manager-istio-csr -f values.yaml --version 0.7.1 jetstack/cert-manager-istio-csr

step 6. repeat step 1 to step 5 for istio-cluster2. at step 4 change clusterID: "istio-cluster1" to clusterID: "istio-cluster2"

Step 6 Deploy istio

this step we install istio base, pull values.yaml for istio-control plane, configure CA Adress and multi cluster settings, and deploy istiod using Helm

step 1. install istio base

helm upgrade --install istio-base -n istio-system --version 1.20.0 istio/base

step 2. pull values.yaml from https://github.com/istio/istio/blob/1.20.0/manifests/charts/istio-control/istio-discovery/values.yaml

step 3. change line 385 to:

  caAddress: "cert-manager-istio-csr.istio-system.svc:443"

step 4. change line 453–461 to:

  multiCluster:
    # Set to true to connect two kubernetes clusters via their respective
    # ingressgateway services when pods in each cluster cannot directly
    # talk to one another. All clusters should be using Istio mTLS and must
    # have a shared root CA for this model to work.
    enabled: true
    # Should be set to the name of the cluster this installation will run in. This is required for sidecar injection
    # to properly label proxies
    clusterName: "istio-cluster1"

step 5. change line 428 to:

meshID: "mesh1"

step 6. change line 474 to:

  network: "network1"

step 7. install istiod

helm upgrade --install istiod --values values.yaml --version 1.20.0 -n istio-system istio/istiod

step 8. verify CA managed by istio-csr & cert-manager. make sure status READY=True

kubectl get certificate -n istio-system

NAME     READY   SECRET       AGE
istiod   True    istiod-tls   2m51s

step 9. describe certificate to get more information about certificate.

kubectl describe certificate istiod -n istio-system

Name:         istiod
Namespace:    istio-system
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: cert-manager-istio-csr
              meta.helm.sh/release-namespace: istio-system
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-04-20T15:21:53Z
  Generation:          1
  Resource Version:    395475
  UID:                 9dbb288b-be14-47af-ae19-30cf00ac26b2
Spec:
  Common Name:  istiod.istio-system.svc
  Dns Names:
    istiod.istio-system.svc
  Duration:  1h0m0s
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   vault
  Private Key:
    Algorithm:             RSA
    Rotation Policy:       Always
    Size:                  2048
  Renew Before:            30m0s
  Revision History Limit:  1
  Secret Name:             istiod-tls
  Uris:
    spiffe://cluster.local/ns/istio-system/sa/istiod-service-account
Status:
  Conditions:
    Last Transition Time:  2024-04-20T15:21:53Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-04-20T16:21:53Z
  Not Before:              2024-04-20T15:21:23Z
  Renewal Time:            2024-04-20T15:51:53Z
  Revision:                1

step 10. verify Certificate Request istiod-1status APPROVED & READY is True.

kubectl get certificaterequests -n istio-system

NAME              APPROVED   DENIED   READY   ISSUER   REQUESTOR                                           AGE

istiod-1          True                True    vault    system:serviceaccount:cert-manager:cert-manager     6m57s

step 11. describe to get more information about certificate request istiod-1

kubectl describe certificaterequests -n istio-system istiod-1

Name:         istiod-1
Namespace:    istio-system
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  cert-manager.io/certificate-name: istiod
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: istiod-494jm
              meta.helm.sh/release-name: cert-manager-istio-csr
              meta.helm.sh/release-namespace: istio-system
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2024-04-20T15:21:53Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  istiod
    UID:                   9dbb288b-be14-47af-ae19-30cf00ac26b2
  Resource Version:        395466
  UID:                     80038a26-d500-48bf-b766-1c862b69ac7e
Spec:
  Duration:  1h0m0s
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-7bfbbd5f46-tw448
    authentication.kubernetes.io/pod-uid:
      c2ce26d4-ea3a-44ce-81a3-d4fe4a1031c6
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:   cert-manager.io
    Kind:    Issuer
    Name:    vault
  Request:   LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQzdUQ0NBZFVDQVFBd0lqRWdNQjRHQTFVRUF4TVhhWE4wYVc5a0xtbHpkR2x2TFhONWMzUmxiUzV6ZG1NdwpnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDK0J6aEdKdGdjb0RJUnMydm9sZndvCkttOGtRY2pYTGNBTDY1bHFYc09ZeEd3QlE0bGwzeUlyK1Bjc0NNY2RScXJ1QVJibWRxYTFVT0hRUHpOOGFEbXgKTUsyaHNKaVpVbWg0VXJRb0tMSC9ScmczVWlFaG44SzYrYWdRaEtTQ1A0OHp1VnJiTFhNU2ozaC84eENPbjlnYQpXSXNTRW11ek13UTFGVkpJT1FhL3IrU1c3dWN3cTg3MnFuTDB6VGpIZDAwdFVWNUNveXhMVURrZ3F5bGF6Ym9kCjFDN0w4d1RnYjdmeHU2V2tKR3N5RldjSXMraUE3MFJHbytnTGpuV3pNZXZNZGR4TUJtNFVTVkV3dEN0R2g0UFkKNXZwUGx4MVhZQVpsbkRhZHFLd2xoRWRQWjVxV1B2TS9JYVNodEZmR0JVR3Uwc0FKa0EveTNLV3hsOUdDbmJiTApBZ01CQUFHZ2dZVXdnWUlHQ1NxR1NJYjNEUUVKRGpGMU1ITXdaQVlEVlIwUkJGMHdXNElYYVhOMGFXOWtMbWx6CmRHbHZMWE41YzNSbGJTNXpkbU9HUUhOd2FXWm1aVG92TDJOc2RYTjBaWEl1Ykc5allXd3Zibk12YVhOMGFXOHQKYzNsemRHVnRMM05oTDJsemRHbHZaQzF6WlhKMmFXTmxMV0ZqWTI5MWJuUXdDd1lEVlIwUEJBUURBZ1dnTUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmdSN0FWR09LRTQwZEpIeFRZUkwyT2wra2ltckx6OENLQk91OFgzbUVVClFPUEZuSHhVZ2RCbitRQ2ZKSEx4US9iTFpXNys4SXR1RmVXZXNjSVpsZ2tqbVQ3OGlNaEtSRmR5eUovcXJJUnQKUHFNQUN3dm1XMFpBck04WnJmUythYStFNmlqZXRnMGVNeWlSMWZyY0VHSHEyb2p6TlJ1SWo1d0E1ZU90OEVtKwpGZVdxQmhtWVk4ejRCRFdBU1hSL0pQK0VxR2lGNHlmd1BmcVpvTVR5WWRZN1VKaHdoZEFwZ09ydk1LVXNYSFBmCkRxcnB6Ym5ocjczYkZtOE43TEdLUndoSnZkandlTHQrZWpjOStXY3ZQbEd0em1lV2pUUXlYcnQ4bTBaajNOYWcKeG1zSENseEo4S0w5VVZFejdEMHpHYWtiY1RteHBnVE5WZkswNFp2ejIvNTEKLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
  UID:       f66ab8a5-81a1-47c1-82a3-a91b336ea2aa
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Ca:           LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQakNDQWlhZ0F3SUJBZ0lVUjhIMzVrVDB2NUxIdXhKYmRuZWRFRHVWSTJBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF4TU9lVzkxY21SdmJXRnBiaTVqYjIwd0hoY05NalF3TkRJd01UVXhNekEzV2hjTgpNelF3TkRFNE1UVXhNek0zV2pBWk1SY3dGUVlEVlFRREV3NTViM1Z5Wkc5dFlXbHVMbU52YlRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxrQTRVM1l1cmg2VVBRTDB4eFpuSzd2N3NYVk5pSWwKTW4wZkdYOXhhbVpieUJXUHFKQUNhWEUxd2RVUWtoM0VCdElMeTlJT1FFZjhmRDgyWnI2akxNcWRTdWxVZ3UyZQo2ejFjQ3gxWng2aDBOV0VGbTJpcXc0Z1N6R1dZWmkyanRHSXdRS2MwMlV0QXBpQ2o0OHMvejdya1FDZFRpRW5aCjlwbXgySjUxUjZodEliNHFEN0pLWnBqUDhMaVgrYjF2VVE3eUMwWFJxVGlCa0h6RkNRTjBOcVh5Mnc1bVVjSEsKTC92M1dPZ2hWMS8zQTltdDF0MGl2M3QxNDZPODViaW1uRDdKKy9jdUZpSk5XMW1ieXd1R0hvZXduaHUzbXd4MgprRXU5TWh2SVJGdmlEWFBwa3BDYVVscW5wL29mRGZCbm1qRjdKdEVDaXk0K3hwQXZYUFo1ei9VQ0F3RUFBYU4rCk1Id3dEZ1lEVlIwUEFRSC9CQVFEQWdFR01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZBZmIKNXVGK3ZmZzZaODd4SjlqZHI5dEtuSXBVTUI4R0ExVWRJd1FZTUJhQUZBZmI1dUYrdmZnNlo4N3hKOWpkcjl0SwpuSXBVTUJrR0ExVWRFUVFTTUJDQ0RubHZkWEprYjIxaGFXNHVZMjl0TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCCkFRQjUrMDY3TWlTVjRlWHpUeW43bDQ3a09rT3pXcXdUNTVWOHVKcVdVSExERmNlWWJ1U2x0eHBkUUp2QktPMDIKZGZqcjloamk5UjIrd3Q5eVBWZmEwajB1U0dmaktyUERaK1djZkZBOE9MU3dCenBaMzhjMVB5eGpBbndQN0JzRgpBc1ZWdGM3dHlsak9yYm1RKy9vTG8xWS9BOXpzbER0REVkaDBBN3BRenhtQlBYeExFTmkyVmlJQWZ4dlhBYWxPCnk5UXNPZFBBejYxRDZ6SS9UYU9KTStYRWtrVFdqVU15cXNlRzEyLzVhTGNZUUt0L2xicE9KcXZTeHJRbXlDdVUKcFljdkRUeFlkblNoc2JZekpNVGUwWVQxeGpaZWsvaGFrM3BjMUUvVDluUUdXaXQrWGU1dkFadHBPc0xDMldtMAo2OVhkR1BCWUxWZFVybnFqd1VMakRKZEsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  Certificate:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURyekNDQXBlZ0F3SUJBZ0lVRS9sV05YY2tYYzc1cFFFV3RXS3RQekdGN2Jjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF4TWJlVzkxY21SdmJXRnBiaTVqYjIwZ2FXNTBaWEp0WldScFlYUmxNQjRYRFRJMApNRFF5TURFMU1qRXlNMW9YRFRJME1EUXlNREUyTWpFMU0xb3dJakVnTUI0R0ExVUVBeE1YYVhOMGFXOWtMbWx6CmRHbHZMWE41YzNSbGJTNXpkbU13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQysKQnpoR0p0Z2NvRElSczJ2b2xmd29LbThrUWNqWExjQUw2NWxxWHNPWXhHd0JRNGxsM3lJcitQY3NDTWNkUnFydQpBUmJtZHFhMVVPSFFQek44YURteE1LMmhzSmlaVW1oNFVyUW9LTEgvUnJnM1VpRWhuOEs2K2FnUWhLU0NQNDh6CnVWcmJMWE1TajNoLzh4Q09uOWdhV0lzU0VtdXpNd1ExRlZKSU9RYS9yK1NXN3Vjd3E4NzJxbkwwelRqSGQwMHQKVVY1Q295eExVRGtncXlsYXpib2QxQzdMOHdUZ2I3Znh1NldrSkdzeUZXY0lzK2lBNzBSR28rZ0xqbld6TWV2TQpkZHhNQm00VVNWRXd0Q3RHaDRQWTV2cFBseDFYWUFabG5EYWRxS3dsaEVkUFo1cVdQdk0vSWFTaHRGZkdCVUd1CjBzQUprQS95M0tXeGw5R0NuYmJMQWdNQkFBR2pnZGd3Z2RVd0RnWURWUjBQQVFIL0JBUURBZ09vTUIwR0ExVWQKSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFkQmdOVkhRNEVGZ1FVMW04M2xMU2llbVJuWmVVRgpXWWZPeXFPYUhwNHdId1lEVlIwakJCZ3dGb0FVT1JpVTV0Q3ZLZkxpZXJyNEQxWTFzN2VHL2lFd1pBWURWUjBSCkJGMHdXNElYYVhOMGFXOWtMbWx6ZEdsdkxYTjVjM1JsYlM1emRtT0dRSE53YVdabVpUb3ZMMk5zZFhOMFpYSXUKYkc5allXd3Zibk12YVhOMGFXOHRjM2x6ZEdWdEwzTmhMMmx6ZEdsdlpDMXpaWEoyYVdObExXRmpZMjkxYm5RdwpEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS3Q3KzdFOEt0d3crYlpiTDlIU2lySzV0Uy9UWFVTaEgxblZ6ZitzCmxPNi8xc3AyR0lnVkhLNzEvR1VGSkZEbVZaRUpQUE9JbDhJd2lweUYzanhKWUZRZ1RzS3FleStlNzJCakorYUYKaUtHWndDNnNvOXAvdzY2SlYyVHBCSnIrRXErcWdONHBtTTN0T2lSaVJsY2p5N2VrOThqYU11YXZOMjlzZGhEawpCcE9XZW1wVGFZRW44NEpMUFRmekJiNDhQOFpsbVlWaTk0MFhaVG5icndCUTRraStTSFVlMHRlZnFuaEVxUkduCnBZaTFGS05mSjRVc3gyVExraDdMZWZ2ZXRwL0RiR2VEdUlTTlZSNDk2enlNckNWTytDTzhIdUcxdURFcjJLOEUKN3FkeUVlL0VoSkZrY0ptTEJOSXVzYVFxN2hqNis4TFBWSFVXdnBUU3F4N1hIRUk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURuakNDQW9hZ0F3SUJBZ0lVS051WVdMdDV4ZENIWDVseWs4UzJNWmpWWW04d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF4TU9lVzkxY21SdmJXRnBiaTVqYjIwd0hoY05NalF3TkRJd01UVXhOekkwV2hjTgpNamt3TkRFNU1UVXhOelUwV2pBbU1TUXdJZ1lEVlFRREV4dDViM1Z5Wkc5dFlXbHVMbU52YlNCcGJuUmxjbTFsClpHbGhkR1V3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzl0RmU3M2NiSW50RTEKZ2orQXE2QXJCc25NTi9YQlF5d1JBUXR0d093bklEVlF0ZTdkNzd3U0JQaFQzZmhFR0Rrc3BRTDBiZGFTTVBSYgpSTmtaeWsxNFlEY0p1ekFWWEl2eWdKQ2t2UDVyL2hUSFk5MFR2bS9QRDgvTFRzckVENC9NUHJQYWxqQWpZRitLCi9qR3JIOXl4Ry9wRXVJUHRWdWZoUzcxdE9WakUrSnRYRytJNk5iZkVwTUR1Z3lsM1NVRTd4bTM4dmZ1ZVlQNzEKM1ZRL1A0ZU1aSVNkUTM5Q2xUMVhSUWo3RU9KUWlCcUFBZU5pLzZoMlpyUXJ5ZmtGMkNBdHk3Y1NoeGVCZFBTSApQRndjckduMTZ6bjdrL1lOci9PdmlyZmRHdmpZVFNQbVpJSGdPTEZ4c3BMU2F6TE1wTXF4MUxibE94TnpWbTBJCmVyYThGRksxQWdNQkFBR2pnZEF3Z2Mwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUIKQWY4d0hRWURWUjBPQkJZRUZEa1lsT2JRcnlueTRucTYrQTlXTmJPM2h2NGhNQjhHQTFVZEl3UVlNQmFBRkFmYgo1dUYrdmZnNlo4N3hKOWpkcjl0S25JcFVNRGtHQ0NzR0FRVUZCd0VCQkMwd0t6QXBCZ2dyQmdFRkJRY3dBb1lkCk1USTNMakF1TUM0eE9qZ3lNREF2ZGpFdmNHdHBYM0p2YjNRdlkyRXdMd1lEVlIwZkJDZ3dKakFrb0NLZ0lJWWUKTVRJM0xqQXVNQzR4T2pneU1EQXZkakV2Y0d0cFgzSnZiM1F2WTNKc01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUE2NHZWWjhUdFF0N1drb2ZwVlNFU29rdkNtbU1ObzBWZlFuS1pQMzVaeVdPY2h1NUdRSUtyajVraFpubzE5CnhXU1NRVkRpM1VIbTFrMDNyVU5COHAzbjYxc2xXazNuK2Z0YlhzWkZxdjdSN3ZNWGJKNlpIK1dmN1VIYkVyR28KMVNaSXl5NnJ1TVRhSzE4WHFuVGVKTWlzZzVMSXNrSDcyaGFobXExWUd4THVTWUNBTGlIY2pGemMyWUg4ZnJ2UApJamduN2xzSHQvV0ZJcWtYTmxPWjJkUUx2Z0VzS1cvYzJXYjVrcksveENLMGgwRmdlTmswdDJSQXpUZHJscDVmCldGc2VDUXFuZDZWTGdwL29JN1lqT1NWU0F3eGsyUE41VWNPaExVbnptTlJ2eGhRQWVySEFKdmRvSUhlZnlSTmcKRm9yZUpFR2ZBS1lMKzY3UUp1bnlheUlPCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  Conditions:
    Last Transition Time:  2024-04-20T15:21:53Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2024-04-20T15:21:53Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason              Age        From                                                Message       
  ----    ------              ----       ----                                                -------       
  Normal  WaitingForApproval  <invalid>  cert-manager-certificaterequests-issuer-ca          Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  <invalid>  cert-manager-certificaterequests-issuer-venafi      Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  <invalid>  cert-manager-certificaterequests-issuer-acme        Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  <invalid>  cert-manager-certificaterequests-issuer-selfsigned  Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  <invalid>  cert-manager-certificaterequests-issuer-vault       Not signing CertificateRequest until it is Approved
  Normal  cert-manager.io     <invalid>  cert-manager-certificaterequests-approver           Certificate request has been approved by cert-manager.io
  Normal  CertificateIssued   <invalid>  cert-manager-certificaterequests-issuer-vault       Certificate fetched from issuer successfully

step 12. repeat step 1 to step 11. just change at step 4 istio-cluster1 toistio-cluster2

Step 7 Configuring istio multicluster

in this step, Configuring istio multicluster using network model Single network. Make sure pod to pod ip on both cluster can connect to each other. on gcp by default ip pod can be accessed directly by instance with same vpc. don’t forget to enable firewall to allow ingress to pod ip.

step 1. install istioctl

step 2. create remote secret from istio-cluster2. change <your istio-cluster1 context> and <your istio-cluster2 context> to your related context.

istioctl create-remote-secret \
    --context="<your istio-cluster1 context" \
    --name=istio-cluster1 | \
    kubectl apply -f - --context="<your istio-cluster2 context"

step 3. create remote secret from istio-cluster1. change <your istio-cluster1 context> and <your istio-cluster2 context> to your related context.

istioctl create-remote-secret \
    --context="<your istio-cluster2 context" \
    --name=istio-cluster2 | \
    kubectl apply -f - --context="<your istio-cluster1 context"

Step 8 Testing Istio Multicluster

step 1. create namespace sample with injected istio sidecar on both clusters.

kubectl create ns sample
kubectl label ns sample istio-injection=enabled

step 2. Create HelloWorld service in both clusters

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l service=helloworld -n sample

step 3. Deploy HelloWorld V1 apps to istio-cluster1

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v1 -n sample

step 4. Deploy HelloWorld V2 apps to istio-cluster2

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v2 -n sample

step 5. Deploy sleep apps to both clusters

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml \
-n sample

step 6. Send multiple request from sleep pod to helloworld service. test to both clusters.

for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld:5000/hello; done

make sure that the helloworld version should toggle between v1 and v2

Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq

conclusion

make sure each cluster have same root CA. though each cluster have different intermediate CA, but all intermediate CA signed by same root CA.
single network model don’t need istio-gateway to communicate pod to pod because they can directly connect to pod with ip pod. if pod to pod can’t access with pod ip, make sure your network propoerly like firewall or routing. if you using GKE just add firewall to allow ingress with destination your pod ip.
multi primary mode deployment istio have istio control plane on each cluster.