Istio Traffic Mirroring Between Clusters

A few months ago, I was working on improving the reliability of a critical service. One of the biggest concerns for the service was around deployments. How could we improve the confidence that changes going out to production were not going to cause issues? Sure, we had unit tests and integration tests but that only verified what we know and expect. Since our environment uses Istio, we decided to take advantage of Istio traffic mirroring production traffic into our staging environment, what better way to find problems with new software releases than actual production traffic?

We have multiple environments consisting of development, staging, and production. Each environment is a separate Kubernetes cluster that runs Istio. In this write-up, we will mirror production traffic to the staging environment. The domain we use for production is authz.example.com and the staging domain is authz-staging.example.com.

When Istio mirrors traffic, the FQDN appends the suffix -shadow e.g. example.com-shadow. In our example, authz.example.com mirrors the staging environment as authz.example.com-shadow.

In order to configure traffic mirroring, we need to create a DestinationRule, a ServiceEntry, and a VirtualService. The DestinationRule is needed in our environment because our ingress enforces TLS traffic e.g. all HTTP traffic is permanently redirected to HTTPS.

VirtualService

On the authz-staging.example.com cluster, we need to create/modify a VirtualService that includes the FQDN that is going to be mirrored with the suffix -shadow. Our example uses authz.example.com-shadow.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: authz-staging-example-com
  namespace: example
spec:
  hosts:
    - authz-staging.example.com
    - authz.example.com-shadow
  http:
  - name: 
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: example
        port:
            number: 80

On the authz.example.com cluster, we need to modify VirtualService to specify where we want to mirror our production traffic and what percentage of production traffic we want to mirror.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: authz-example-com
  namespace: example
spec:
  hosts:
    - authz.example.com
  http:
  - name: 
    match:
    - uri:
        prefix: /
    mirror:
      host: authz-staging.example.com
      port:
        number: 443
    mirrorPercentage:
      value: 100
    route:
    - destination:
        host: example
        port:
            number: 80

DestinationRule

The DestinationRule is only required on the authz.example.com cluster. This rule ensures that mirrored traffic is encrypted for the authz-staging.example.com domain, which is required because we enforce HTTPS traffic only.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: authz-example-com
  namespace: example
spec:
  host: authz-staging.example.com
  trafficPolicy:
    tls:
      mode: SIMPLE

If you are not enforcing HTTPS traffic, you can omit this rule.

ServiceEntry

The ServiceEntry is only required on the authz.example.com cluster because authz-staging.example.com is running on another Kubernetes cluster.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: authz-example-com-https
  namespace: example
spec:
  hosts:
    - authz-staging.example.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

After you have added the DestinationRule and ServiceEntry, and modified the VirtualService, you should see traffic flowing from your production cluster to your staging cluster.

The Istio documentation has a great demo app that you can configure for traffic mirroring [1], and I would recommend going through that demo too.