Istio Service Mesh by Practical Example

This post includes:

• Installation of Fundamental Components
• Observability/Traceability
• Service Discovery/Routing
• Ingress/Load Balancing
• AuthN/AuthZ for Zero Trust Model
• Network Resilience: Circuit Breaker and Others
• Egress
• Cross-Cluster Mesh
• Performance/Load Testing
• When Isito meets Knative

Most the demo images used in this post is built with Tanzu Build Server.

In this article, I also use google online boutique source code as example. If working in air-gapped env, build the image on your own or use following script to migrate from gcr.io to local registry.

$ git clone ...
$ cd release
$ cat migrate.sh
IMGLI=$(grep "gcr.io/google-samples" kubernetes-manifests.yaml | sed -e 's/^[ \t]*//' | cut -d ' ' -f 2)
DSTR='harbor.run.haas-481.pez.vmware.com/microservices'
for img in $IMGLI
do 
  docker pull $img
  IMGN=$(echo $img | cut -d '/' -f 4)
  echo $IMGN
  docker tag $img $DSTR/$IMGN
  docker push $DSTR/$IMGN
done

Installation

1. Install cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.0/cert-manager.yaml

Create cluster issuer

$ openssl genrsa -out ca.key 2048
$ export COMMON_NAME="haas-495.pez.vmware.com"
$ openssl req -x509 -new -nodes -key ca.key -subj "/CN=${COMMON_NAME}" -days 3650 -reqexts v3_req -extensions v3_ca -out ca.crt
$ kubectl create secret tls tls-ca-secret -n cert-manager --cert ./ca.crt --key ./ca.key
$ cat << EOF | kubectl apply -f -
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: tls-selfsigned-issuer
    spec:
      ca:
        secretName: tls-ca-secret
    EOF

2. Install Istio

$ curl -L https://istio.io/downloadIstio | sh -
$ cd istio-1.12.0.   ##depends on the latest version
$ sudo cp bin/* /usr/local/bin
$ istioctl install --set profile=default -y

3. Install Prometheus

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm pull bitnami/kube-prometheus
$ tar xvfz kube-prometheus.tgz
$ cd kube-prometheus
$ vim values.yaml
prometheus.ingress.enable: true
prometheus.ingress.annotations
      kubernetes.io/ingress.class: istio
      cert-manager.io/cluster-issuer: tls-selfsigned-issuer

It’s a little weird that cert-manager will create the tls certificate secret in monitoring namespace. Whereas istio ingress gateway reads certificate in istio-system namespace. Therefore, we need to create certificate in istio-system namespacde manually.

$ cat << EOF | k apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kube-prometheus
    helm.sh/chart: kube-prometheus-6.4.1
  name: prometheus.haas-481.pez.vmware.com-tls
  namespace: istio-system
spec:
  dnsNames:
  - prometheus.haas-481.pez.vmware.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: tls-selfsigned-issuer
  secretName: prometheus.haas-481.pez.vmware.com-tls
  usages:
  - digital signature
  - key encipherment
EOF

Create ServiceMonitor to monitor the Istio control plane and PodMonitor Envoy proxies

$ cd istio-1.12.0 
$ kubectl apply -f samples/addons/extras/prometheus-operator.yaml

4. Install Grafana

same as prometheus

Import istio dashboard into grafana:

7639 11829 7636 7630 7645

5. Install Kiali
Install kiali operator

$ helm repo add kiali https://kiali.org/helm-charts
$ helm repo update
$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator 

Install kiali server

$ cat << EOF | kubectl apply -f -
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  annotations:
    meta.helm.sh/release-name: kiali-operator
    meta.helm.sh/release-namespace: kiali-operator
  labels:
    app: kiali-operator
    app.kubernetes.io/instance: kiali-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kiali-operator
    app.kubernetes.io/part-of: kiali-operator
    app.kubernetes.io/version: v1.43.0
    helm.sh/chart: kiali-operator-1.43.0
    version: v1.43.0
  name: kiali
  namespace: istio-system
spec:
  deployment:
    accessible_namespaces:
    - '**'
    service_type: LoadBalancer
  external_services:
    prometheus:
    url: http://promethus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/
  server:
    port: 8080
    web_fqdn: kiali.haas-495.pez.vmware.com
EOF

Get access token:

k get secret $(k get secret -n istio-system | grep kiali-service-account-token | cut -d ' ' -f 1) -o=jsonpath="{.data.token}" -n istio-system |base64 -d

Observability/Traceability

Service Discovery/Routing

Istio takes advantage of the Kubernetes list of services and endpoints to populate and maintain its Service Registry. It uses this Service Registry to instruct the Envoy proxies as to where to direct the application’s traffic.

When there are multiple instances of the same microservice available, by default, Envoy uses the Istio Service Registry to send the traffic to each instance in a round-robin fashion. Istio provides a Traffic Management API that can be used to add, alter, or remove specific behaviors by adjusting the Kubernetes Custom Resource Definitions (CRDs) that define them

Virtual services, along with destination rules, are the key building blocks of Istio’s traffic routing functionality. A virtual service lets you configure how requests are routed to a service within an Istio service mesh, building on the basic connectivity and discovery provided by Istio and your platform. Each virtual service consists of a set of routing rules that are evaluated in order, letting Istio match each given request to the virtual service to a specific real destination within the mesh. Your mesh can require multiple virtual services or none depending on your use case.

Let’s deploy microservices:

Create a new namespace and add a namespace label to instruct Istio to automatically inject Envoy sidecar proxies when you deploy your application later

$ kubectl create ns ms-workload
$ kubectl label namespace ms-workload istio-injection=enabled

Deploy microservice:

$ cd release
$ vim kubernetes-manifests.yaml ##uncomment DISABLE_PROFILER for service recommendationservice
$ kubectl apply -f kubernetes-manifests.yaml -n ms-workload

Add routing:

##Create tls certicate for front-end service
$ cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mall.haas-495.pez.vmware.com-cert
  namespace: istio-system
spec:
  dnsNames:
  - mall.haas-495.pez.vmware.com
  issuerRef:
    kind: ClusterIssuer
    name: app-selfsigned-issuer
  secretName: mall.haas-495.pez.vmware.com-tls
  usages:
  - digital signature
  - key encipherment
##Modify gateway to add https access
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: frontend-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80       #####for test only
      name: http
      protocol: HTTP
    hosts:
    - mall.haas-495.pez.vmware.com
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - mall.haas-495.pez.vmware.com
    tls:
      credentialName: mall.haas-495.pez.vmware.com-tls
      mode: SIMPLE ##TLS terminates here.
##Virtual servcies
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-ingress
spec:
  hosts:
  - mall.haas-495.pez.vmware.com
  gateways:
  - frontend-gateway
  http:
  - route:
    - destination:
        host: frontend
        port:
          number: 80

Kiali graph for the application

Ingress/Load Balancing

AuthN/AuthZ for Zero Trust Model

Authentication refers to identity: who is this service? who is this end-user? and can I trust that they are who they say they are?

One benefit of using Istio that it provides uniformity for both service-to-service and end user-to-service authentication. Istio abstracts away authentication from your application code, by tunneling all service-to-service communication through the Envoy sidecar proxies. And by using a centralized Public-Key Infrastructure, Istio provides consistency to make sure authentication is set up properly across your mesh. Further, Istio allows you to adopt mTLS on a per-service basis, or easily toggle end-to-end encryption for your entire mesh.

Default mTLS behavior

Starting in Istio 1.5, the default Istio mTLS behavior is “auto.” This means that pod-to-pod traffic will use mutual TLS by default, but pods will still accept plain-text traffic(Permissive mode) — for instance, from pods in a different namespace that are not injected with the Istio proxy. This is evident by checking Kiali graph — from display dropdown, checking security, as below:

we see a lock icon on the edges in the graph, indicating that traffic is encrypted/mTLS.

Authentication policies

Istio uses yaml file do define policy. A policy might be applied to :

1. workload-specific
2. namespace-wide
3. mesh-wide

Let’s define a namespace-wide policy

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "default"
spec:
  mtls:
    mode: STRICT

End-User JWT Authentication

Request authentication policies specify the values needed to validate a JSON Web Token (JWT). These values include, among others, the following:

• The location of the token in the request
• The issuer or the request
• The public JSON Web Key Set (JWKS)

Let’s create and enforce end-user (“origin”) authentication for the frontend service

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: frontend
  action: ALLOW
  rules:
  - from:
    - source:
       requestPrincipals:
         ["testing@secure.istio.io/testing@secure.istio.io"]

Authorization

Unlike authentication, which refers to the “who,” authorization refers to the “what”, or: what is this service or user allowed to do?

By default, requests between Istio services (and between end-users and services) are allowed by default. We can then enforce authorization for one or many services using an AuthorizationPolicy custom resource. An authorization policy includes a selector, an action, and a list of rules:

• The selector field specifies the target of the policy
• The action field specifies whether to allow or deny the request
• The rules specify when to trigger the action
• The from field in the rules specifies the sources of the request
• The to field in the rules specifies the operations of the request
• The when field specifies the conditions needed to apply the rule

Let’s put this into action, by only allowing requests to the frontend that have a specific HTTP header (hello:world):

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "frontend"
  namespace: default
spec:
  selector:
    matchLabels:
      app: frontend
  rules:
  - when:
    - key: request.headers[hello]
      values: ["world"]

Request without corresponding header:

kubectl exec $(kubectl get pod -l app=productcatalogservice -o jsonpath={.items..metadata.name}) -c istio-proxy \
-- curl http://frontend:80/ -o /dev/null -s -w '%{http_code}\n'

403

Request with header:

kubectl exec $(kubectl get pod -l app=productcatalogservice -o jsonpath={.items..metadata.name}) -c istio-proxy \
-- curl --header "hello:world" http://frontend:80 -o /dev/null -s -w '%{http_code}\n'

200

Circuit Breaker Pattern

Istio provides opt-in failure recovery and fault injection features to help your applications operate reliably, ensuring that the service mesh can tolerate failing nodes and preventing localized failures from cascading to other nodes. These features include:

• Retries and Timeouts.
• Circuit breakers
• Health checks
• Outlier detection
• Fault injection.

In Istio, we need to create a destination rule to configure circuit breaking for a target service. In his blog post, “Demystifying Istio Circuit Breaking” , Vikas Kumar uses following parameter:

tcp.maxConnections: Maximum number of HTTP1 /TCP connections to a destination host. Default 2³²-1.

http.http1MaxPendingRequests: Maximum number of pending HTTP requests to a destination. Default 2³²-1.

http.http2MaxRequests: Maximum number of requests to a backend. Default 2³²-1.

I follow this post and write server and client respectively to test mentioned scenarios in Vikas’s post.

Server:

Server.go
package main

import (
    "fmt"
    "log"
    "net/http"
    "time"
)

func main() {
    http.HandleFunc("/index", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(5 * time.Second)
	fmt.Fprintf(w, "Hello")
	})
    log.Fatal(http.ListenAndServe(":8081", nil))
}
------
server-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-server
spec:
  selector:
    matchLabels:
      app: go-web-server
  replicas: 1
  template:
    metadata:
      labels:
        app: go-web-server
    spec:
      containers:
      - name: go-web-server
        image: harbor.haas-495.pez.vmware.com/app/go-web-server:v1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: go-web-service
spec:
  selector:
    app: go-web-server
  ports:
  - protocol: "TCP"
    port: 80
    targetPort: 8081
  type: ClusterIP

Client:

Client.go
package main

import (
    "fmt"
    "net/http"
    "time"
)

func workerRoute(url string) {
    _start := time.Now()
    resp, err := http.Get(url)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    _end := time.Now()

    fmt.Printf("STATUS: %s START: %02d:%02d:%02d, END: %02d:%02d:%02d, TIME: %d \n",
        resp.Status,
        _start.Hour(),
        _start.Minute(),
        _start.Second(),
        _end.Hour(),
        _end.Minute(),
        _end.Second(),
        _end.Sub(_start))
}

func main() {
    time.Sleep(30 * time.Second)
    for {
        switch time.Now().Second() {
        case 0, 20, 40:
            for i := 0; i < 10; i++ {
                go workerRoute("http://go-web-service/index")
            }
            time.Sleep(time.Second)
        default:
            time.Sleep(time.Second)
            continue
        }
    }
}
---
Client-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-client
spec:
  selector:
    matchLabels:
      app: go-web-client
  replicas: 4
  template:
    metadata:
      labels:
        app: go-web-client
    spec:
      containers:
      - name: go-web-client
        image: harbor.haas-495.pez.vmware.com/app/go-web-client:v1.0
        imagePullPolicy: Always

Test result is similar to Vikas’s conclusion

On the client-side:

Each client proxy applies the limit independently. If the limit is 100, then each client proxy can have 100 outstanding requests before they apply local throttling. If there are 80 clients calling a destination service, there can be a maximum of 8000 outstanding requests overall.

The limit on the client proxy is for the destination service overall, not for individual replicas of the destination service. The throttling will still happen at 100, even if there are, say 200, active pods of the destination service.

On the destination side:

Each destination service proxy also applies the limit. If there are 50 active pods of the service, each pod can have a maximum of 100 outstanding requests from client proxies before applying throttling and returning 503.

Egress

Istio egress gateway offers several advantages:

• Allocate dedicated cluster nodes to deploy egress gateway. All the outbound traffic must and only pass through theses nodes. Security team could enforce specific hardened measures on these externally facing nodes(pool) , plus enhanced configuration on firewall, so as to ensure high security level.
• Use Taints and Tolerations on egress nodes to prevent workload from deploying on theses nodes. That way, we can isolate internal workload from external world.

Isito provides threes ways to access external services in a mesh:

1. Allow the Envoy proxy to pass requests through to services that are not configured inside the mesh.
2. Configure service entries to provide controlled access to external services.
3. Completely bypass the Envoy proxy for a specific range of IPs.

The second approach lets us use all of the same Istio service mesh features for calls to services inside or outside of the cluster. We can take advantage of these mesh features to monior and control external traffic flow.

1. Configure dedicated nodes:

$ kubectl label node node1 egress-node=true
$ kubectl taint node node1 egress-key=egress-value:NoSchedule

2. Modify istio manifest, adding below items:

egressGateways:
    - name: istio-egressgateway
      enabled: true
      k8s:
        nodeSelector:
          egress-node: "true"
        tolerations:
        - key: "egress-key"
          operator: "Equal"
          values: "egress-values"
          effect: "NoSchedule"    

3. Deploy egress gateway

$ istioctl manifest apply -f mesh.yaml

4. Define a ServiceEntry for http://mall.haas-495.pez.vmware.com

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mall
spec:
  hosts:
  - mall.haas-495.pez.vmware.com
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
EOF

5. Define gateway and rule for the outbound traffic:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: mall
spec:
  hosts:
    - mall.haas-495.pez.vmware.com
  http:
  - timeout: 3s
    route:
      - destination:
          host: mall.haas-495.pez.vmware.com
        weight: 100
EOF

This way, we use kubectl to set a 3s timeout on calls to the mall.haas-495.pez.vmware.com external service.

Cross-Cluster Mesh

In this part, we will expand istio mesh across multiple kubernetes clusters.

There are several deployment scenarios in terms of multicluster istio mesh: multi-primary, primary-remote and their variants for different networks. In this tutorial, I will deploy a primary-remote mesh on different networks. This type of configuration has several advantages. Firstly, it support overlapped ip address across clusters. Secondly, we don’t have to pay more attention to network toils, such as firewall configuration, for examples.

Primary and remote clusters on separate networks from istio.io

In this tutorial, I create two k8s clusters, with a context name of control and remote respectively(or use).

$ kubectl --context control create ns istio-system
$ kubectl --context remote create ns istio-system

1. Configure trust

A multicluster service mesh deployment requires that establishing trust between all clusters in the mesh. By default the Istio CA generates a self-signed root certificate and key and uses them to sign the workload certificates. To protect the root CA key, you should use a root CA which runs on a secure machine offline, and use the root CA to issue intermediate certificates to the Istio CAs that run in each cluster. An Istio CA can sign workload certificates using the administrator-specified certificate and key, and distribute an administrator-specified root certificate to the workloads as the root of trust.

The following graph demonstrates the recommended CA hierarchy in a mesh containing two clusters.

CA Hierarchy from istio.io

In this tutorial, I use tools from istio install package to generate root CA and local CA. In production environment, we can use certificates generated by PKI system or security team.

$ cd $ISTIO_HOME_DIRECTORY
$ mkdir -p certs
$ pushed certs
$ make -f ../tools/certs/Makefile.selfsigned.mk root-ca
$ make -f ../tools/certs/Makefile.selfsigned.mk control-cacerts
$ make -f ../tools/certs/Makefile.selfsigned.mk remote-cacerts
$ kubectl --context control create secret generic cacerts -n istio-system \
    --from-file=control-cacerts/ca-cert.pem \
    --from-file=control-cacerts/ca-key.pem \
    --from-file=control-cacerts/root-cert.pem \
    --from-file=control-cacerts/cert-chain.pem
$ kubectl --context remote create secret generic cacerts -n istio-system \
    --from-file=remote-cacerts/ca-cert.pem \
    --from-file=remote-cacerts/ca-key.pem \
    --from-file=remote-cacerts/root-cert.pem \
    --from-file=remote-cacerts/cert-chain.pem
$ popd

**In Istio, Citadel is the service that is responsible for signing and distributing certificates to all Envoy proxies in the service mesh. Citadel uses a secret named cacerts , if any, to initiate the system. If needed rotate the certificates, just recreate this secret and restart istiod pod and application deployments(rolling out or canary)

2. Configure control cluster as a primary

2.1 Set cluster’s network

$ kubectl --context=control get namespace istio-system && \   kubectl --context=control label namespace istio-system topology.istio.io/network=net-control

2.2. Create the Istio configuration for control

$ cat meshinfo/istio-control.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: control
  namespace: istio-system
spec:
  values:
    global:
      meshID: mesh1
      network: net-control
      multiCluster:
        clusterName: control
  components:
    ingressGateways:
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          topology.istio.io/network: net-control
        enabled: true
        k8s:
          env:
            # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
            - name: ISTIO_META_ROUTER_MODE
              value: "sni-dnat"
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: net-control
          service:
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017

2.3 Install the Istio control plane on the control cluster (press y to proceed):

$ istioctl --context control manifest apply -f meshinfo/istio-control.yaml

2.4 Ensure that all Istio deployments are running:

kubectl --context control get pods -n istio-system

2.5 Wait for the east-west gateway to be assigned an external IP address:

kubectl --context=control get svc istio-eastwestgateway -n istio-system

2.6 Expose the control plane (istiod Service) in the control cluster so that services in the remote cluster are able to access service discovery:

$ kubectl apply --context=control -n istio-system -f samples/multicluster/expose-istiod.yaml

This command creates a Gateway and a VirtualService spec for the istiod Service to be exposed through the istio-eastwestgateway

2.7 Expose services in cluster control

$ kubectl --context=control apply -n istio-system -f \     samples/multicluster/expose-services.yaml

3. Configure remote cluster

3.1 Set cluster’s network

$ kubectl --context=remote get namespace istio-system && \   kubectl --context=remote label namespace istio-system topology.istio.io/network=net-remote

3.2 Enable API Server Access to remote cluster

Before configuring Istio on the remote cluster, you must give the control plane in the control cluster access to the API Server in the remote cluster. This will do the following:

• Enables the control plane to authenticate connection requests from workloads running in the remote cluster. Without API Server access, the control plane will reject the requests.
• Enables discovery of service endpoints running in the remote cluster.

$ istioctl x create-remote-secret \
--context=remote \
--name=remote | \
kubectl apply -f - --context=control

This command will:

• Create a service account named istio-reader-service-account in remote cluster
• Create two cluster roles and bind to istio-reader-service-account in remote cluster
• Create a secret named istio-remote-secret-{remote cluster name} in control cluster

3.2. Create the Istio configuration for remote

$ cat meshinfo/istio-remote.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: remote
      network: net-remote
      remotePilotAddress: 10.212.133.59
      caAddress: istiod.istio-system.svc:15012
  components:
    ingressGateways:
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          topology.istio.io/network: net-remote
        enabled: true
        k8s:
          env:
            # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
            - name: ISTIO_META_ROUTER_MODE
              value: "sni-dnat"
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: net-remote
          service:
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017

**we can get remotePilotAddress by:

$ kubectl --context=control -n istio-system \
get svc istio-eastwestgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}'

**If caAddress isn’t specified, x.509 error maybe be thrown.

3.3 Install Istio resources on the remote cluster (press y to proceed):

$ istioctl --context remote manifest apply -f meshinfo/istio-remote.yaml

3.4 Ensure that all Istio deployments are running:

$ kubectl --context remote get pods -n istio-system

3.5 Wait for the east-west gateway to be assigned an external IP address:

$ kubectl --context=remote get svc istio-eastwestgateway -n istio-system

3.6 Expose services running in the remote cluster through the istio-eastwestgateway:

$ kubectl --context=remote apply -n istio-system -f \     samples/multicluster/expose-services.yaml

4. Redeploying the Online Boutique app as follow:

Diagram from Google Cloud

5. Access the app, but the page doesn’t work at all.

Why?

6. Check proxy status: OK

$ istioctl proxy status

7. Check the endpoints the frontend service has: OK

istioctl --context control pc endpoint frontend-59b7d8554c-g64dp.ms-workload | grep ms-workload

istioctl --context remote pc endpoint cartservice-7b45d78f99-zbcdh.ms-workload | grep ms-workload

All of information seems ok. But when checking frontend pod’s log, I found the app returned “not found error” when looked up cart service on kube-dns server. It seemed reasonabl esince cart service is located on another cluster(remote). So can frontend discovery remote services from istio services registry rather than form kubernetes CoreDNS? This is where “DNS Proxying ” shines.

8. DNS proxying

In addition to capturing application traffic, Istio can also capture DNS requests to improve the performance and usability of your mesh. When proxying DNS, all DNS requests from an application will be redirected to the sidecar, which stores a local mapping of domain names to IP addresses. If the request can be handled by the sidecar, it will directly return a response to the application, avoiding a roundtrip to the upstream DNS server. Otherwise, the request is forwarded upstream following the standard /etc/resolv.conf DNS configuration.

While Kubernetes provides DNS resolution for Kubernetes Services out of the box, any custom ServiceEntrys will not be recognized. With this feature, ServiceEntry addresses can be resolved without requiring custom configuration of a DNS server. For Kubernetes Services, the DNS response will be the same, but with reduced load on kube-dns and increased performance.

Let’s modify our IstioOperator yaml files — both for control cluster and for remote, as follow:

kind: IstioOperator
metadata:
  name: control
  namespace: istio-system
spec:
  meshConfig:
    defaultConfig:
      proxyMetadata:
        # Enable basic DNS proxying
        ISTIO_META_DNS_CAPTURE: "true"
        # Enable automatic address allocation, optional
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"
  values:
    global:
      meshID: mesh1
      network: net-control
      multiCluster:
        clusterName: control
  components:
    ingressGateways:
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          topology.istio.io/network: net-control
        enabled: true
        k8s:
          env:
            # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
            - name: ISTIO_META_ROUTER_MODE
              value: "sni-dnat"
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: net-control
          service:
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017

*Add high-lighted part, and redeploy the istio system.

Let’s access out web site again:

It works!

From Kiali graph

We can see recommendation service and cart service are on remote cluster.

Some tips:

1. Adjust log level for pod during troubleshooting

istioctl pc log pod-name --level:debug

2. Install isito with custom registry or work around docker hub rate limit

Solution 1: add imagepullsecrets for istiod or other service account
Solution 2: add global configuration
apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
spec:
 values:
   global:
     imagePullSecrets:
     — myregistrykey

Performance/Load Testing

Istio uses fortio as performance/load testing tool.

The test environment is two Fortio pods (one client, one server), set to communicate over HTTP1, using mutual TLS authentication.

git clone https://github.com/istio/tools.git
cd tools/perf/benchmark
export NAMESPACE=app-workload
export INTERCEPTION_MODE=REDIRECT
export ISTIO_INJECT=false
export LOAD_GEN_TYPE=fortio
export DNS_DOMAIN=local
./setup_test.sh

Let’s test performance without istio sidecar injection.

Test scritp:

CLIENTPOD=$(kubectl get pod -n app-workload | grep client | cut -d ' '  -f 1)
for (( k = 2; k < 9000; k=k*2 )); do
  k -n app-workload exec $CLIENTPOD -c captured -- fortio load \
   -jitter=False \
   -c $k \
   -qps 0 \
   -t 60s \
   -a http://fortioserver:8080/echo\?size\=1024
done

Test result:

We can get a result of 35987 qps for default deployment.

Next, let’s enable istio inject and redeploy the envrionemnt

export NAMESPACE=app-workload
export INTERCEPTION_MODE=REDIRECT
export ISTIO_INJECT=true
export LOAD_GEN_TYPE=fortio
export DNS_DOMAIN=local
./setup_test.sh

Run test script again, but we can only get results with a highest qps of 3.79k.

Remove sidecar for fortioclient, still only 5k or so qps:

It’s obvious that fortio server sidecar needed to be optimized in order to improve the throughput.

According to Envoy threading model:

In istio proxy, there’s configuration item “concurrency” specifying The number of worker threads to run. If unset, this will be automatically determined based on CPU requests/limits. If set to 0, all cores on the machine will be used. Default is 2 worker threads.

Let’s tweak concurrency to “8” which is the vCPU number of node.

$ kubectl edit deployment fortioserver -n app-workload
add:
 annotation:
  proxy.istio.io/config: |-
    concurrency: 8
    proxyStatsMatcher:
      inclusionRegexps:
        - "listener.*.downstream_cx_total"
        - "listener.*.downstream_cx_active"

We also add metrics for per-worker for listeners in previous command.

Next, we add a custom dashboard in Grafana monitoring connection distribution per worker thread.

{__name__=~”envoy_listener_worker_.+_downstream_cx_active”}

Right now, let’s do test again. We can see the qps value rising to over 10k. But it’s still far from qps value without sidecar at all, although pod cpu consumption reached more than 90% of node total cpus. Maybe, further optimization needed.

As for connection distribution, we can see from below diagram that the connections are nearly evenly distributed among worker threads.

When Isito meets Knative

Knative Serving system pods, such as the activator and autoscaler components, require access to your deployed Knative services. If you have configured additional security features, such as Istio’s authorization policy, you must enable access to your Knative service for these system pods.

1. Istio must be used for cluster Knative Ingress.
2. Istio sidecar injection must be enabled

kubectl label namespace knative-serving istio-injection=enabled

3. Grant service access privilege to knative system namespace

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-serving-tests
 namespace: serving-tests
spec:
 action: ALLOW
 rules:
 - from:
   - source:
      namespaces: ["serving-tests", "knative-serving"]

4. Grant metrics collection and health check privilege to knative system ns

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allowlist-by-paths
  namespace: serving-tests
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        paths:
        - /metrics   # The path to collect metrics by system pod.
        - /healthz   # The path to probe by system pod.

From istio.io

References

1. https://thenewstack.io/sailing-faster-with-istio-part-1/
2. https://kiali.io/docs/installation/installation-guide/install-with-helm/
3. https://cloud.google.com/architecture/building-gke-multi-cluster-service-mesh-with-istio-shared-control-plane-disparate-networks
4. https://cloud.google.com/architecture/service-meshes-in-microservices-architecture
5. “Service Mesh for Mere Mortals” by Bruce Basil Mathews
6. https://istio.io/latest/docs/ops/deployment/deployment-models/#network-model
7. https://istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/
8. https://tech.olx.com/demystifying-istio-circuit-breaking-27a69cac2ce4
9. https://istio.io/latest/docs/tasks/traffic-management/egress/egress-control/