Istio multicluster with istio-csr + cert-manager + vault PKI
Goal
Certificate published by Vault PKI, using cert-manager for handling cert from vault, and istio-csr to request cert from cert-manager and distribute cert to all istio-proxy (sidecar) & istiod. and can communicate across cluster kubernetes.
Prerequisite
- vault v1.16.1
- istio v1.20.0
- cert-manager v1.13.2
- istio-csr 0.7.1
Architecture
multicluster (multi primary) single network
Introduction
Root and Intermediate Certificate Authority (CA)
The root CA is the top-level certificate authority that signs the intermediate CA certificate. It establishes the trust hierarchy within the PKI.
Vault
Vault can be configured to act as a CA (Certificate Authority) for Istio multicluster deployments. It can manage both the root CA and intermediate CA certificates, providing a secure foundation for issuing certificates within and across clusters.
cert-manager
cert-manager can be configured to issue certificates for Istio components, such as Ingress Gateways, using Vault or Let’s Encrypt as the certificate issuer. This includes defining certificate requests, policies, and renewal settings.
istio-csr
istio-csr can be utilized to handle certificate signing requests for services that need to communicate securely between clusters. This includes generating certificates for Istio gateways and services that span multiple clusters.
Step 1 Deploy Vault
step 1. Add helm repo hashicorp
helm repo add hashicorp https://helm.releases.hashicorp.com
step 2. deploy vault using helm chart
helm upgrade --install --version 0.27.0 vault hashicorp/vault
step 3. init vault to get unseal key. keep this unseal key.
kubectl exec -it vault-0 -- vault operator init
and you get unseal key 1–5, also root token for login to vault.
Unseal Key 1: szzicGp+99ZIT0bJ5QVriy+ItI319HzYiRpVUg8+Qmq0
Unseal Key 2: gWcDh4cF24D+YRlDDjjtScBF5+jLzdTh3AYdqjYyieJZ
Unseal Key 3: w1LheoTZjMZxUmU4j1SAnZXc+r0hnbziLz022LdT4DTr
Unseal Key 4: 7vz8OSU9eS7Xr32QEgG6s5JiwZSYvTH2vI7LJE5wDiMw
Unseal Key 5: xpbBRO/u0OjuiN+YaywblMcuKlPMWnmMY5mFMWg5v5DG
Initial Root Token: hvs.OjlXv5Yz80QAtpeEJoEKPSpA
step 4. unseal with key above just key 1 to key 3.
kubectl exec -it vault-0 -- vault operator unseal szzicGp+99ZIT0bJ5QVriy+ItI319HzYiRpVUg8+Qmq0
kubectl exec -it vault-0 -- vault operator unseal gWcDh4cF24D+YRlDDjjtScBF5+jLzdTh3AYdqjYyieJZ
kubectl exec -it vault-0 -- vault operator unseal w1LheoTZjMZxUmU4j1SAnZXc+r0hnbziLz022LdT4DTr
Step 2 Create Vault PKI root CA
in this step, we set up the Vault PKI root CA, enabling the pki secrets engine, tuning it for certificate issuance, generating the root CA, and configuring the CA and CRL distribution points.
step 1. install vault-cli and export env VAULT_TOKEN=<YOUR ROOT TOKEN GENERATE FROM VAULT INIT>
& VAULT_ADDR=<YOUR VAULT ADDRESS>
note: you can running command `kubectl port-forward svc/vault 8200:8200` and your vault address is 127.0.0.1:8200
export VAULT_TOKEN=<VAULT_TOKEN>
export VAULT_ADDR=<your vault address>
step 2. Enable pki secrets engine to path pki_root
vault secrets enable -path=pki_root pki
step 3. tune pki to issue certificate with Maximum TTL 87600 hours or 10 year.
vault secrets tune -max-lease-ttl=87600h pki_root
step 4. generate root CA with file name CA_cert.crt
- common_name is string, you can fill anyting. but i recommend to fill your domain.
- ttl is time-to-live your expired certificate. i set ttl same with maximum TTL on point 3.
vault write -field=certificate pki_root/root/generate/internal \
common_name="yourdomain.com" ttl=87600h > CA_cert.crt
step 5. configure CA and CRL change vault.yourdomain.com:8200
to your url vault.
vault write pki_root/config/urls \
issuing_certificates="vault.yourdomain.com:8200/v1/pki_root/ca" \
crl_distribution_points="vault.yourdomain.com:8200/v1/pki_root/crl"
Step 3 Create Vault PKI intermediate CA
This step involves creating and intermediate PKI for the istio cluster, enabling the pki secrets engine, tuning it, generating a CSR for the intermediate CA, signing it with the root CA, creating a certificate chain, and setting the signed intermediate CA.
step 1. Enable pki secrets engine to path pki_int1_istio-cluster1
vault secrets enable -path=pki_int1_istio-cluster1 pki
step 2. tune pki to issue certificate with Maximum TTL 43800 hours or 5 year.
vault secrets tune -max-lease-ttl=43800h pki_int1_istio-cluster1
step 3. generate intermediate CA CSR with file name pki_intermediate1_istio_cluster1.csr
- common_name is string, you can fill anyting. but i recommend to fill your domain.
- ttl is time-to-live your expired certificate. i set ttl same with maximum TTL on point 3.
vault write -format=json pki_int1_istio-cluster1/intermediate/generate/internal \
common_name="yourdomain.com intermediate 1" \
| jq -r '.data.csr' > pki_intermediate1_istio_cluster1.csr
step 4. sign intermediate CA CSR with the root CA and generate file `intermediate1_istio_cluster1.cert.pem`
vault write -format=json pki_root/root/sign-intermediate csr=@pki_intermediate1_istio_cluster1.csr \
format=pem ttl="43800h" \
| jq -r '.data.certificate' > intermediate1_istio_cluster1.cert.pem
step 5. create certificate chain to combines the intermediate CA and root CA into a single file.
cat intermediate1_istio_cluster1.cert.pem > intermediate1_istio_cluster1.chain.pem
cat CA_cert.crt >> intermediate1_istio_cluster1.chain.pem
step 6. set signed intermediate CA.
vault write pki_int1_istio-cluster1/intermediate/set-signed certificate=@intermediate1_istio_cluster1.chain.pem
step 7. create role for pki_int1_istio-cluster1
with name istio-ca-istio-cluster1
for specifying various constraints such as allowed_domains, max_ttl,etc.
vault write pki_int1_istio-cluster1/roles/istio-ca-istio-cluster1 \
allowed_domains=istio-ca \
allow_any_name=true \
enforce_hostnames=false \
require_cn=false \
allowed_uri_sans="spiffe://*" \
max_ttl=72h
step 8. repeat step 1 to step 7 for istio-cluster2. and rename:
- pki_int1_istio-cluster1 to pki_int1_istio-cluster2
- pki_intermediate1_istio_cluster1.csr to pki_intermediate1_istio_cluster2.csr
- intermediate1_istio_cluster1.cert.pem to intermediate1_istio_cluster2.cert.pem
- intermediate1_istio_cluster1.chain.pem to intermediate1_istio_cluster2.chain.pem
- istio-ca-istio-cluster1 to istio-ca-istio-cluster2
Step 4 Deploy and Configure cert-manager
this step we deploy cert-manager using Helm, deploy CRDs, create a secret for the vault token, and configure cert-manager to communicate with vault for issuing certificates.
step 1. add helm repo and update.
helm repo add jetstack https://charts.jetstack.io
helm repo update
step 2. deploy crds cert-manager.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.crds.yaml
step 3. create namespace cert-manager
& istio-system
kubectl create ns cert-manager
kubectl create ns istio-system
step 4. deploy cert-manager.
helm upgrade --install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.13.2
step 5. create secret for token vault
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: cert-manager-vault-token
namespace: istio-system
data:
token: <your vault token with base64 encode>
EOF
step 6. add issuer to vault
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: vault
namespace: istio-system
spec:
vault:
server: http://vault.yourdomain.com:8200
path: pki_int1_istio-cluster1/sign/istio-ca-istio-cluster1
auth:
tokenSecretRef:
name: cert-manager-vault-token
key: token
EOF
step 6. repeat step 1 to step 6 for istio-cluster2. at step 5 change path pki_int1_istio-cluster1/sign/istio-ca-istio-cluster1
to pki_int1_istio-cluster2/sign/istio-ca-istio-cluster2
Step 5 Deploy istio-csr
this step we pull values.yaml for istio-csr, modify configurations for issuer, certificate DNS names, and istio cluster id, and deploy istio-csr using Helm, ensuring seamless integration with istio components.
step 1. pull values.yaml from https://github.com/cert-manager/istio-csr/blob/main/deploy/charts/istio-csr/values.yaml
step 2. change line 93–102 to:
issuer:
# -- Issuer name set on created CertificateRequests for both istio-csr's
# serving certificate and incoming gRPC CSRs.
name: vault
# -- Issuer kind set on created CertificateRequests for both istio-csr's
# serving certificate and incoming gRPC CSRs.
kind: Issuer
# -- Issuer group name set on created CertificateRequests for both
# istio-csr's serving certificate and incoming gRPC CSRs.
group: cert-manager.io
step 3. change line 115 to:
certificateDNSNames:
- cert-manager-istio-csr.istio-system.svc
step 4. change line 139 to:
server:
# -- The istio cluster ID to verify incoming CSRs.
clusterID: "istio-cluster1"
step 5. deploy istio-csr
helm upgrade --install -n istio-system cert-manager-istio-csr -f values.yaml --version 0.7.1 jetstack/cert-manager-istio-csr
step 6. repeat step 1 to step 5 for istio-cluster2. at step 4 change clusterID: "istio-cluster1"
to clusterID: "istio-cluster2"
Step 6 Deploy istio
this step we install istio base, pull values.yaml for istio-control plane, configure CA Adress and multi cluster settings, and deploy istiod using Helm
step 1. install istio base
helm upgrade --install istio-base -n istio-system --version 1.20.0 istio/base
step 2. pull values.yaml from https://github.com/istio/istio/blob/1.20.0/manifests/charts/istio-control/istio-discovery/values.yaml
step 3. change line 385 to:
caAddress: "cert-manager-istio-csr.istio-system.svc:443"
step 4. change line 453–461 to:
multiCluster:
# Set to true to connect two kubernetes clusters via their respective
# ingressgateway services when pods in each cluster cannot directly
# talk to one another. All clusters should be using Istio mTLS and must
# have a shared root CA for this model to work.
enabled: true
# Should be set to the name of the cluster this installation will run in. This is required for sidecar injection
# to properly label proxies
clusterName: "istio-cluster1"
step 5. change line 428 to:
meshID: "mesh1"
step 6. change line 474 to:
network: "network1"
step 7. install istiod
helm upgrade --install istiod --values values.yaml --version 1.20.0 -n istio-system istio/istiod
step 8. verify CA managed by istio-csr & cert-manager. make sure status READY=True
kubectl get certificate -n istio-system
NAME READY SECRET AGE
istiod True istiod-tls 2m51s
step 9. describe certificate to get more information about certificate.
kubectl describe certificate istiod -n istio-system
Name: istiod
Namespace: istio-system
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: cert-manager-istio-csr
meta.helm.sh/release-namespace: istio-system
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2024-04-20T15:21:53Z
Generation: 1
Resource Version: 395475
UID: 9dbb288b-be14-47af-ae19-30cf00ac26b2
Spec:
Common Name: istiod.istio-system.svc
Dns Names:
istiod.istio-system.svc
Duration: 1h0m0s
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: vault
Private Key:
Algorithm: RSA
Rotation Policy: Always
Size: 2048
Renew Before: 30m0s
Revision History Limit: 1
Secret Name: istiod-tls
Uris:
spiffe://cluster.local/ns/istio-system/sa/istiod-service-account
Status:
Conditions:
Last Transition Time: 2024-04-20T15:21:53Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2024-04-20T16:21:53Z
Not Before: 2024-04-20T15:21:23Z
Renewal Time: 2024-04-20T15:51:53Z
Revision: 1
step 10. verify Certificate Request istiod-1
status APPROVED & READY is True.
kubectl get certificaterequests -n istio-system
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
istiod-1 True True vault system:serviceaccount:cert-manager:cert-manager 6m57s
step 11. describe to get more information about certificate request istiod-1
kubectl describe certificaterequests -n istio-system istiod-1
Name: istiod-1
Namespace: istio-system
Labels: app.kubernetes.io/managed-by=Helm
Annotations: cert-manager.io/certificate-name: istiod
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: istiod-494jm
meta.helm.sh/release-name: cert-manager-istio-csr
meta.helm.sh/release-namespace: istio-system
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2024-04-20T15:21:53Z
Generation: 1
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: istiod
UID: 9dbb288b-be14-47af-ae19-30cf00ac26b2
Resource Version: 395466
UID: 80038a26-d500-48bf-b766-1c862b69ac7e
Spec:
Duration: 1h0m0s
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-7bfbbd5f46-tw448
authentication.kubernetes.io/pod-uid:
c2ce26d4-ea3a-44ce-81a3-d4fe4a1031c6
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: vault
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQzdUQ0NBZFVDQVFBd0lqRWdNQjRHQTFVRUF4TVhhWE4wYVc5a0xtbHpkR2x2TFhONWMzUmxiUzV6ZG1NdwpnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDK0J6aEdKdGdjb0RJUnMydm9sZndvCkttOGtRY2pYTGNBTDY1bHFYc09ZeEd3QlE0bGwzeUlyK1Bjc0NNY2RScXJ1QVJibWRxYTFVT0hRUHpOOGFEbXgKTUsyaHNKaVpVbWg0VXJRb0tMSC9ScmczVWlFaG44SzYrYWdRaEtTQ1A0OHp1VnJiTFhNU2ozaC84eENPbjlnYQpXSXNTRW11ek13UTFGVkpJT1FhL3IrU1c3dWN3cTg3MnFuTDB6VGpIZDAwdFVWNUNveXhMVURrZ3F5bGF6Ym9kCjFDN0w4d1RnYjdmeHU2V2tKR3N5RldjSXMraUE3MFJHbytnTGpuV3pNZXZNZGR4TUJtNFVTVkV3dEN0R2g0UFkKNXZwUGx4MVhZQVpsbkRhZHFLd2xoRWRQWjVxV1B2TS9JYVNodEZmR0JVR3Uwc0FKa0EveTNLV3hsOUdDbmJiTApBZ01CQUFHZ2dZVXdnWUlHQ1NxR1NJYjNEUUVKRGpGMU1ITXdaQVlEVlIwUkJGMHdXNElYYVhOMGFXOWtMbWx6CmRHbHZMWE41YzNSbGJTNXpkbU9HUUhOd2FXWm1aVG92TDJOc2RYTjBaWEl1Ykc5allXd3Zibk12YVhOMGFXOHQKYzNsemRHVnRMM05oTDJsemRHbHZaQzF6WlhKMmFXTmxMV0ZqWTI5MWJuUXdDd1lEVlIwUEJBUURBZ1dnTUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmdSN0FWR09LRTQwZEpIeFRZUkwyT2wra2ltckx6OENLQk91OFgzbUVVClFPUEZuSHhVZ2RCbitRQ2ZKSEx4US9iTFpXNys4SXR1RmVXZXNjSVpsZ2tqbVQ3OGlNaEtSRmR5eUovcXJJUnQKUHFNQUN3dm1XMFpBck04WnJmUythYStFNmlqZXRnMGVNeWlSMWZyY0VHSHEyb2p6TlJ1SWo1d0E1ZU90OEVtKwpGZVdxQmhtWVk4ejRCRFdBU1hSL0pQK0VxR2lGNHlmd1BmcVpvTVR5WWRZN1VKaHdoZEFwZ09ydk1LVXNYSFBmCkRxcnB6Ym5ocjczYkZtOE43TEdLUndoSnZkandlTHQrZWpjOStXY3ZQbEd0em1lV2pUUXlYcnQ4bTBaajNOYWcKeG1zSENseEo4S0w5VVZFejdEMHpHYWtiY1RteHBnVE5WZkswNFp2ejIvNTEKLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
UID: f66ab8a5-81a1-47c1-82a3-a91b336ea2aa
Username: system:serviceaccount:cert-manager:cert-manager
Status:
Ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQakNDQWlhZ0F3SUJBZ0lVUjhIMzVrVDB2NUxIdXhKYmRuZWRFRHVWSTJBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF4TU9lVzkxY21SdmJXRnBiaTVqYjIwd0hoY05NalF3TkRJd01UVXhNekEzV2hjTgpNelF3TkRFNE1UVXhNek0zV2pBWk1SY3dGUVlEVlFRREV3NTViM1Z5Wkc5dFlXbHVMbU52YlRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxrQTRVM1l1cmg2VVBRTDB4eFpuSzd2N3NYVk5pSWwKTW4wZkdYOXhhbVpieUJXUHFKQUNhWEUxd2RVUWtoM0VCdElMeTlJT1FFZjhmRDgyWnI2akxNcWRTdWxVZ3UyZQo2ejFjQ3gxWng2aDBOV0VGbTJpcXc0Z1N6R1dZWmkyanRHSXdRS2MwMlV0QXBpQ2o0OHMvejdya1FDZFRpRW5aCjlwbXgySjUxUjZodEliNHFEN0pLWnBqUDhMaVgrYjF2VVE3eUMwWFJxVGlCa0h6RkNRTjBOcVh5Mnc1bVVjSEsKTC92M1dPZ2hWMS8zQTltdDF0MGl2M3QxNDZPODViaW1uRDdKKy9jdUZpSk5XMW1ieXd1R0hvZXduaHUzbXd4MgprRXU5TWh2SVJGdmlEWFBwa3BDYVVscW5wL29mRGZCbm1qRjdKdEVDaXk0K3hwQXZYUFo1ei9VQ0F3RUFBYU4rCk1Id3dEZ1lEVlIwUEFRSC9CQVFEQWdFR01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZBZmIKNXVGK3ZmZzZaODd4SjlqZHI5dEtuSXBVTUI4R0ExVWRJd1FZTUJhQUZBZmI1dUYrdmZnNlo4N3hKOWpkcjl0SwpuSXBVTUJrR0ExVWRFUVFTTUJDQ0RubHZkWEprYjIxaGFXNHVZMjl0TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCCkFRQjUrMDY3TWlTVjRlWHpUeW43bDQ3a09rT3pXcXdUNTVWOHVKcVdVSExERmNlWWJ1U2x0eHBkUUp2QktPMDIKZGZqcjloamk5UjIrd3Q5eVBWZmEwajB1U0dmaktyUERaK1djZkZBOE9MU3dCenBaMzhjMVB5eGpBbndQN0JzRgpBc1ZWdGM3dHlsak9yYm1RKy9vTG8xWS9BOXpzbER0REVkaDBBN3BRenhtQlBYeExFTmkyVmlJQWZ4dlhBYWxPCnk5UXNPZFBBejYxRDZ6SS9UYU9KTStYRWtrVFdqVU15cXNlRzEyLzVhTGNZUUt0L2xicE9KcXZTeHJRbXlDdVUKcFljdkRUeFlkblNoc2JZekpNVGUwWVQxeGpaZWsvaGFrM3BjMUUvVDluUUdXaXQrWGU1dkFadHBPc0xDMldtMAo2OVhkR1BCWUxWZFVybnFqd1VMakRKZEsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURyekNDQXBlZ0F3SUJBZ0lVRS9sV05YY2tYYzc1cFFFV3RXS3RQekdGN2Jjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF4TWJlVzkxY21SdmJXRnBiaTVqYjIwZ2FXNTBaWEp0WldScFlYUmxNQjRYRFRJMApNRFF5TURFMU1qRXlNMW9YRFRJME1EUXlNREUyTWpFMU0xb3dJakVnTUI0R0ExVUVBeE1YYVhOMGFXOWtMbWx6CmRHbHZMWE41YzNSbGJTNXpkbU13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQysKQnpoR0p0Z2NvRElSczJ2b2xmd29LbThrUWNqWExjQUw2NWxxWHNPWXhHd0JRNGxsM3lJcitQY3NDTWNkUnFydQpBUmJtZHFhMVVPSFFQek44YURteE1LMmhzSmlaVW1oNFVyUW9LTEgvUnJnM1VpRWhuOEs2K2FnUWhLU0NQNDh6CnVWcmJMWE1TajNoLzh4Q09uOWdhV0lzU0VtdXpNd1ExRlZKSU9RYS9yK1NXN3Vjd3E4NzJxbkwwelRqSGQwMHQKVVY1Q295eExVRGtncXlsYXpib2QxQzdMOHdUZ2I3Znh1NldrSkdzeUZXY0lzK2lBNzBSR28rZ0xqbld6TWV2TQpkZHhNQm00VVNWRXd0Q3RHaDRQWTV2cFBseDFYWUFabG5EYWRxS3dsaEVkUFo1cVdQdk0vSWFTaHRGZkdCVUd1CjBzQUprQS95M0tXeGw5R0NuYmJMQWdNQkFBR2pnZGd3Z2RVd0RnWURWUjBQQVFIL0JBUURBZ09vTUIwR0ExVWQKSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFkQmdOVkhRNEVGZ1FVMW04M2xMU2llbVJuWmVVRgpXWWZPeXFPYUhwNHdId1lEVlIwakJCZ3dGb0FVT1JpVTV0Q3ZLZkxpZXJyNEQxWTFzN2VHL2lFd1pBWURWUjBSCkJGMHdXNElYYVhOMGFXOWtMbWx6ZEdsdkxYTjVjM1JsYlM1emRtT0dRSE53YVdabVpUb3ZMMk5zZFhOMFpYSXUKYkc5allXd3Zibk12YVhOMGFXOHRjM2x6ZEdWdEwzTmhMMmx6ZEdsdlpDMXpaWEoyYVdObExXRmpZMjkxYm5RdwpEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS3Q3KzdFOEt0d3crYlpiTDlIU2lySzV0Uy9UWFVTaEgxblZ6ZitzCmxPNi8xc3AyR0lnVkhLNzEvR1VGSkZEbVZaRUpQUE9JbDhJd2lweUYzanhKWUZRZ1RzS3FleStlNzJCakorYUYKaUtHWndDNnNvOXAvdzY2SlYyVHBCSnIrRXErcWdONHBtTTN0T2lSaVJsY2p5N2VrOThqYU11YXZOMjlzZGhEawpCcE9XZW1wVGFZRW44NEpMUFRmekJiNDhQOFpsbVlWaTk0MFhaVG5icndCUTRraStTSFVlMHRlZnFuaEVxUkduCnBZaTFGS05mSjRVc3gyVExraDdMZWZ2ZXRwL0RiR2VEdUlTTlZSNDk2enlNckNWTytDTzhIdUcxdURFcjJLOEUKN3FkeUVlL0VoSkZrY0ptTEJOSXVzYVFxN2hqNis4TFBWSFVXdnBUU3F4N1hIRUk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURuakNDQW9hZ0F3SUJBZ0lVS051WVdMdDV4ZENIWDVseWs4UzJNWmpWWW04d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF4TU9lVzkxY21SdmJXRnBiaTVqYjIwd0hoY05NalF3TkRJd01UVXhOekkwV2hjTgpNamt3TkRFNU1UVXhOelUwV2pBbU1TUXdJZ1lEVlFRREV4dDViM1Z5Wkc5dFlXbHVMbU52YlNCcGJuUmxjbTFsClpHbGhkR1V3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzl0RmU3M2NiSW50RTEKZ2orQXE2QXJCc25NTi9YQlF5d1JBUXR0d093bklEVlF0ZTdkNzd3U0JQaFQzZmhFR0Rrc3BRTDBiZGFTTVBSYgpSTmtaeWsxNFlEY0p1ekFWWEl2eWdKQ2t2UDVyL2hUSFk5MFR2bS9QRDgvTFRzckVENC9NUHJQYWxqQWpZRitLCi9qR3JIOXl4Ry9wRXVJUHRWdWZoUzcxdE9WakUrSnRYRytJNk5iZkVwTUR1Z3lsM1NVRTd4bTM4dmZ1ZVlQNzEKM1ZRL1A0ZU1aSVNkUTM5Q2xUMVhSUWo3RU9KUWlCcUFBZU5pLzZoMlpyUXJ5ZmtGMkNBdHk3Y1NoeGVCZFBTSApQRndjckduMTZ6bjdrL1lOci9PdmlyZmRHdmpZVFNQbVpJSGdPTEZ4c3BMU2F6TE1wTXF4MUxibE94TnpWbTBJCmVyYThGRksxQWdNQkFBR2pnZEF3Z2Mwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUIKQWY4d0hRWURWUjBPQkJZRUZEa1lsT2JRcnlueTRucTYrQTlXTmJPM2h2NGhNQjhHQTFVZEl3UVlNQmFBRkFmYgo1dUYrdmZnNlo4N3hKOWpkcjl0S25JcFVNRGtHQ0NzR0FRVUZCd0VCQkMwd0t6QXBCZ2dyQmdFRkJRY3dBb1lkCk1USTNMakF1TUM0eE9qZ3lNREF2ZGpFdmNHdHBYM0p2YjNRdlkyRXdMd1lEVlIwZkJDZ3dKakFrb0NLZ0lJWWUKTVRJM0xqQXVNQzR4T2pneU1EQXZkakV2Y0d0cFgzSnZiM1F2WTNKc01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUE2NHZWWjhUdFF0N1drb2ZwVlNFU29rdkNtbU1ObzBWZlFuS1pQMzVaeVdPY2h1NUdRSUtyajVraFpubzE5CnhXU1NRVkRpM1VIbTFrMDNyVU5COHAzbjYxc2xXazNuK2Z0YlhzWkZxdjdSN3ZNWGJKNlpIK1dmN1VIYkVyR28KMVNaSXl5NnJ1TVRhSzE4WHFuVGVKTWlzZzVMSXNrSDcyaGFobXExWUd4THVTWUNBTGlIY2pGemMyWUg4ZnJ2UApJamduN2xzSHQvV0ZJcWtYTmxPWjJkUUx2Z0VzS1cvYzJXYjVrcksveENLMGgwRmdlTmswdDJSQXpUZHJscDVmCldGc2VDUXFuZDZWTGdwL29JN1lqT1NWU0F3eGsyUE41VWNPaExVbnptTlJ2eGhRQWVySEFKdmRvSUhlZnlSTmcKRm9yZUpFR2ZBS1lMKzY3UUp1bnlheUlPCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
Conditions:
Last Transition Time: 2024-04-20T15:21:53Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2024-04-20T15:21:53Z
Message: Certificate fetched from issuer successfully
Reason: Issued
Status: True
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitingForApproval <invalid> cert-manager-certificaterequests-issuer-ca Not signing CertificateRequest until it is Approved
Normal WaitingForApproval <invalid> cert-manager-certificaterequests-issuer-venafi Not signing CertificateRequest until it is Approved
Normal WaitingForApproval <invalid> cert-manager-certificaterequests-issuer-acme Not signing CertificateRequest until it is Approved
Normal WaitingForApproval <invalid> cert-manager-certificaterequests-issuer-selfsigned Not signing CertificateRequest until it is Approved
Normal WaitingForApproval <invalid> cert-manager-certificaterequests-issuer-vault Not signing CertificateRequest until it is Approved
Normal cert-manager.io <invalid> cert-manager-certificaterequests-approver Certificate request has been approved by cert-manager.io
Normal CertificateIssued <invalid> cert-manager-certificaterequests-issuer-vault Certificate fetched from issuer successfully
step 12. repeat step 1 to step 11. just change at step 4 istio-cluster1
toistio-cluster2
Step 7 Configuring istio multicluster
in this step, Configuring istio multicluster using network model Single network. Make sure pod to pod ip on both cluster can connect to each other. on gcp by default ip pod can be accessed directly by instance with same vpc. don’t forget to enable firewall to allow ingress to pod ip.
step 1. install istioctl
step 2. create remote secret from istio-cluster2. change <your istio-cluster1 context>
and <your istio-cluster2 context>
to your related context.
istioctl create-remote-secret \
--context="<your istio-cluster1 context" \
--name=istio-cluster1 | \
kubectl apply -f - --context="<your istio-cluster2 context"
step 3. create remote secret from istio-cluster1. change <your istio-cluster1 context>
and <your istio-cluster2 context>
to your related context.
istioctl create-remote-secret \
--context="<your istio-cluster2 context" \
--name=istio-cluster2 | \
kubectl apply -f - --context="<your istio-cluster1 context"
Step 8 Testing Istio Multicluster
step 1. create namespace sample
with injected istio sidecar on both clusters.
kubectl create ns sample
kubectl label ns sample istio-injection=enabled
step 2. Create HelloWorld service in both clusters
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l service=helloworld -n sample
step 3. Deploy HelloWorld V1 apps to istio-cluster1
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v1 -n sample
step 4. Deploy HelloWorld V2 apps to istio-cluster2
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v2 -n sample
step 5. Deploy sleep apps to both clusters
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml \
-n sample
step 6. Send multiple request from sleep pod to helloworld service. test to both clusters.
for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld:5000/hello; done
make sure that the helloworld version should toggle between v1 and v2
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
conclusion
- make sure each cluster have same root CA. though each cluster have different intermediate CA, but all intermediate CA signed by same root CA.
- single network model don’t need istio-gateway to communicate pod to pod because they can directly connect to pod with ip pod. if pod to pod can’t access with pod ip, make sure your network propoerly like firewall or routing. if you using GKE just add firewall to allow ingress with destination your pod ip.
- multi primary mode deployment istio have istio control plane on each cluster.