Istio Series — 3: Performance optimization: Dealing with memory leak — Part I: Customizing metrics

8 min readMay 5, 2024

Hi folks, if you are using istio as service mesh in production with high number of inter-communicating microservices, then this blog is for you.

Istio mentions about performance of their proxy-sidecars as:

The memory consumption of the proxy depends on the total configuration state the proxy holds. A large number of listeners, clusters, and routes can increase memory usage. In a large namespace with namespace isolation enabled, the proxy consumes approximately 50 MB of memory.

But often in huge cluster at some point, unfortunately we may see some pods with sidecar gets OOM killed by Kubernetes. We can also observe that istio-proxy sidecar was the reason behind these crashes. And this issue targets only particular pods, not all pods in the mesh were facing this issue.

I also noticed for the pods with high number of inbound and outbound endpoints, istio-proxy consumes more memory than other pods. but still root-cause is not discovered. I was trying various ways to tune the performance — deployed Sidecar for namespace-isolation. But couldn’t see any difference. I was even seeking help for this issue in istio discuss forum. After days of debugging, I found that istio-metrics emitted from istio-proxy was contributing to this issue. Without metrics emission istio-proxy was not having any issues, it was running fine for days. Thus I understood, we need to optimise istio-metrics before enabling them back in production.

Mitigation Steps

1. Dropping unused labels in each metrics

By default, istio metrics has enormous informative labels, which are mostly unused (atleast for me). For example,

{response_code=”200",reporter=”source”,source_workload=”xxx”,source_workload_namespace=”xxx”,source_principal=”unknown”,source_app=”xxx”,source_version=”unknown”,source_cluster=”xxx”,destination_workload=”unknown”,destination_workload_namespace=”unknown”,destination_principal=”unknown”,destination_app=”unknown”,destination_version=”unknown”,destination_service=”xxx”,destination_service_name=”xxx”,destination_service_namespace=”unknown”,destination_cluster=”unknown”,request_protocol=”http”,response_flags=”-”,grpc_response_status=””,connection_security_policy=”unknown”,source_canonical_service=”xxx”,destination_canonical_service=”unknown”,source_canonical_revision=”latest”,destination_canonical_revision=”latest”,url=”xxx”,request_method=”xxx"}

Of course, I’ve added url and request_method labels in order to capture those informations in emitted metrics. As you can see these many labels contributes to high cardinality, thus creates load on prometheus. In order to reduce cardinality, we need to drop few unused labels. In my case, source and destination principals, clusters, grpc_response_status, versions and canonical_revisions can be dropped, without any harm. Besides we can also optimise few labels like source and destination apps, workloads, canonical_services, workload_namespaces and service_namespaces.

The following envoy filter snippet can be used to drop labels [“source_workload”,”source_principal”,”source_version”,”source_cluster”,”destination_workload”,”destination_principal”,”destination_version”,”destination_service”,”destination_service_name”,”destination_cluster”,”grpc_response_status”,”connection_security_policy”,”source_app”,”destination_app”,”source_canonical_revision”,”destination_canonical_revision”,”destination_service_namespace”] from metric requests_total

             - dimensions:
                request_method: request.method
                destination_canonical_service: "!(request.host.endsWith('.com') ? upstream_peer.labels['app'].value : request.host"
                url: request.headers['x-url-category']
                traffic: "'outbound'"
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
              name: requests_total

2. Drop unused metrics on the whole

We can also drop the metric which is unused. Like request_bytes, response_bytes metrics from istio (Of course, for extensive monitoring we need these metrics to track the Network I/O of our ecosystem), which are again histogram metrics that contributes overhead to the monitoring system.

istio_request_bytes_bucket
istio_request_bytes_count
istio_request_bytes_sum
istio_response_bytes_bucket
istio_response_bytes_count
istio_response_bytes_sum
istio_tcp_received_bytes_total
istio_tcp_sent_bytes_total

We can drop a particular metric by having the following snippet in the envoyfilter.

            - drop: true
              name: request_bytes
            - name: response_bytes
              drop: true

Above snippet will drop request_bytes and response_bytes metrics in istio.

I applied both the above strategies in order to improve the situation. I’ve implemented the metrics optimization by:

Dropping metrics: request_bytes, response_bytes, request_duration_milliseconds (histogram metrics).
Removing the following labels from metrics requests_total: [“source_workload”,”source_principal”,”source_version”,”source_cluster”,”destination_workload”,”destination_principal”,”destination_version”,”destination_service”,”destination_service_name”,”destination_cluster”,”grpc_response_status”,”connection_security_policy”,”source_app”,”destination_app”,”source_canonical_revision”,”destination_canonical_revision”,”destination_service_namespace”]
Since we need latency metric have created a new metric of COUNTER type called, duration_ms_sum. Thus Average latency can be obtained by duration_ms_sum / requests_total.

3. Eliminating costlier metric types

As discussed in point #3 above, I have replaced request_duration_milliseconds metric which is HISTOGRAM type with a new COUNTER metric duration_ms_sum. This is because one histogram metric contains a _sum and _bucket metrics in it, which raises cardinality of your metric system.

Following EnvyFilter snippet helps to “declare” a new custom metric.

            definitions:
            - name: duration_ms_sum
              type: COUNTER
              value: int(response.headers['x-envoy-upstream-service-time'])

This defines duration_ms_sum as a COUNTER metric, with the response time value, captured in x-envoy-upstream-service-time response header.

We also need to “define” the labels of this newly created metric, for that we can use the following snippet:

            - dimensions:
                url: request.headers['x-url-category']
                request_method: request.method
                traffic: "'outbound'"
                response_flags: response.code_details
                source_workload_namespace: node.metadata.NAMESPACE
                source_canonical_service: node.metadata.WORKLOAD_NAME
                request_protocol: request.protocol
                response_code: "(string(response.code) == 'unknown') ? '0' : string(response.code)"
                destination_canonical_service: "!(request.host.endsWith('.com')) ? upstream_peer.labels['app'].value : request.host"
                destination_workload_namespace: "has(upstream_peer.namespace) ? upstream_peer.namespace : 'unknown'"
              name: duration_ms_sum

I hope above dimensions i.e., labels of the metric is self explanatory. This works for outbound traffic, whereas for inbound traffic dimension values varies.

Bonus: We can use the following annotation, to avoid misinterpretaion of newly added tags, to the metric.

sidecar.istio.io/extraStatTags: request_method,url,traffic

The Complete EnvoyFilter code is given below.

NOTE: The given Envoy filter works for Istio version 1.18 and above. For previous versions of Istio typed_config section differs and the value represented in json

Following code works with HTTP_FILTER, which handles http metrics like, requests_total, request_duration_ms_sum, request and response bytes.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  labels:
    install.operator.istio.io/owning-resource: installed-state-1-21-0
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio.io/rev: 1-21-0
    operator.istio.io/component: Pilot
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.21.0
  name: new-metrics-stats
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
      proxy:
        proxyVersion: ^1\.21.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            definitions:
            - name: duration_ms_sum
              type: COUNTER
              value: request.duration
            metrics:
            - drop: true
              name: request_bytes
            - drop: true
              name: response_bytes
            - drop: true
              name: request_duration_milliseconds
            - dimensions:
                destination_canonical_service: '!(request.host.endsWith(".com")) ? upstream_peer.labels["app"].value : request.host'
                destination_workload_namespace: 'has(upstream_peer.namespace) ? upstream_peer.namespace : "unknown"'
                request_method: request.method
                request_protocol: request.protocol
                response_flags: response.code_details
                source_canonical_service: node.metadata.WORKLOAD_NAME
                source_workload_namespace: node.metadata.NAMESPACE
                traffic: '"outbound"'
                url: request.headers['x-url-category']
              name: requests_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
            - dimensions:
                destination_canonical_service: '!(request.host.endsWith(".com")) ? upstream_peer.labels["app"].value : request.host'
                destination_workload_namespace: 'has(upstream_peer.namespace) ? upstream_peer.namespace : "unknown"'
                request_method: request.method
                request_protocol: request.protocol
                response_code: '(string(response.code) == "unknown") ? "0" : string(response.code)'
                response_flags: response.code_details
                source_canonical_service: node.metadata.WORKLOAD_NAME
                source_workload_namespace: node.metadata.NAMESPACE
                traffic: '"outbound"'
                url: request.headers['x-url-category']
              name: duration_ms_sum
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
      proxy:
        proxyVersion: ^1\.21.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            definitions:
            - name: duration_ms_sum
              type: COUNTER
              value: request.duration
            metrics:
            - drop: true
              name: request_bytes
            - drop: true
              name: response_bytes
            - drop: true
              name: request_duration_milliseconds
            - dimensions:
                destination_canonical_service: node.metadata.WORKLOAD_NAME
                destination_workload_namespace: node.metadata.NAMESPACE
                request_method: request.method
                request_protocol: request.protocol
                response_code: '(string(response.code) == "unknown") ? "0" : string(response.code)'
                response_flags: response.code_details
                source_canonical_service: downstream_peer.labels['app'].value
                source_workload_namespace: 'has(downstream_peer.namespace) ? downstream_peer.namespace : "unknown"'
                traffic: '"inbound"'
                url: request.headers['x-url-category']
              name: requests_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
            - dimensions:
                destination_canonical_service: node.metadata.WORKLOAD_NAME
                destination_workload_namespace: node.metadata.NAMESPACE
                request_method: request.method
                request_protocol: request.protocol
                response_code: '(string(response.code) == "unknown") ? "0" : string(response.code)'
                response_flags: response.code_details
                source_canonical_service: downstream_peer.labels['app'].value
                source_workload_namespace: 'has(downstream_peer.namespace) ? downstream_peer.namespace : "unknown"'
                traffic: '"inbound"'
                url: request.headers['x-url-category']
              name: duration_ms_sum
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
      proxy:
        proxyVersion: ^1\.21.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            definitions:
            - name: duration_ms_sum
              type: COUNTER
              value: request.duration
            metrics:
            - drop: true
              name: request_bytes
            - drop: true
              name: response_bytes
            - drop: true
              name: request_duration_milliseconds
            - dimensions:
                destination_canonical_service: upstream_peer.labels["app"].value
                destination_workload_namespace: upstream_peer.namespace
                metric_from: node.metadata.WORKLOAD_NAME
                request_method: request.method
                request_protocol: request.protocol
                response_code: '(string(response.code) == "unknown") ? "0" : string(response.code)'
                response_flags: response.code_details
                source_canonical_service: node.metadata.WORKLOAD_NAME
                source_workload_namespace: node.metadata.NAMESPACE
                traffic: '"outbound"'
                url: request.headers['x-url-category']
              name: requests_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
            - dimensions:
                destination_canonical_service: upstream_peer.labels["app"].value
                destination_workload_namespace: upstream_peer.namespace
                metric_from: node.metadata.WORKLOAD_NAME
                request_method: request.method
                request_protocol: request.protocol
                response_code: '(string(response.code) == "unknown") ? "0" : string(response.code)'
                response_flags: response.code_details
                source_canonical_service: node.metadata.WORKLOAD_NAME
                source_workload_namespace: node.metadata.NAMESPACE
                traffic: '"outbound"'
                url: request.headers['x-url-category']
              name: duration_ms_sum
  priority: -1

Note: Above I’ve used request.duration for accessing latency of the request. There are various other terms which we could easily get confused, like we can also use int(response.headers[‘x-envoy-upstream-service-time’]), but they are not same. Each has slight difference with other. We’ll cover up the differences seperately in another discussion.

Following envoy filter patches NETWORK_FILTER, that emits tcp metrics like, tcp_connections_(opened/closed)_total and tcp_(sent/received)_bytes_total

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  labels:
    install.operator.istio.io/owning-resource: installed-state-1-18-0
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio.io/rev: 1-18-0
    operator.istio.io/component: Pilot
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.18.0
  name: new-metrics-tcp-stats
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
      proxy:
        proxyVersion: ^1\.18.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            metrics:
            - name: tcp_received_bytes_total
              drop: true
            - name: tcp_sent_bytes_total
              drop: true
            - name: tcp_connections_opened_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_canonical_revision
              - source_principal
              - source_app
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_app
              - destination_version
              - destination_service
              - destination_canonical_revision
              - destination_service_name
              - destination_service_namespace
              - destination_cluster
              - connection_security_policy
              dimensions:
                upstream_address: "upstream.address"
                url: "'unknown'"
                traffic: "'outbound'" 
            - name: tcp_connections_closed_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_canonical_revision
              - source_principal
              - source_app
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_app
              - destination_version
              - destination_service
              - destination_canonical_revision
              - destination_service_name
              - destination_service_namespace
              - destination_cluster
              - connection_security_policy
              dimensions:
                upstream_address: "upstream.address"
                url: "'unknown'"
                traffic: "'outbound'" 
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
      proxy:
        proxyVersion: ^1\.18.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            metrics:
            - name: tcp_received_bytes_total
              drop: true
            - name: tcp_sent_bytes_total
              drop: true
            - name: tcp_connections_opened_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_canonical_revision
              - source_principal
              - source_app
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_app
              - destination_version
              - destination_service
              - destination_canonical_revision
              - destination_service_name
              - destination_service_namespace
              - destination_cluster
              - connection_security_policy
              dimensions:
                source_canonical_service: downstream_peer.labels['app'].value
                upstream_address: "upstream.address"
                url: "'unknown'"
                traffic: "'inbound'"
            - name: tcp_connections_closed_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_canonical_revision
              - source_principal
              - source_app
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_app
              - destination_version
              - destination_service
              - destination_canonical_revision
              - destination_service_name
              - destination_service_namespace
              - destination_cluster
              - connection_security_policy
              dimensions:
                source_canonical_service: downstream_peer.labels['app'].value
                upstream_address: "upstream.address"
                url: "'unknown'"
                traffic: "'inbound'"
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
      proxy:
        proxyVersion: ^1\.18.*
    patch:
      operation: INSERT_BEFORE
      value:
        name: istio.stats
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/stats.PluginConfig
          value:
            metrics:
            - name: tcp_received_bytes_total
              drop: true
            - name: tcp_sent_bytes_total
              drop: true
            - name: tcp_connections_opened_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
              dimensions:
                destination_canonical_service: "(cluster_name == 'PassthroughCluster') ? upstream.address : upstream_peer.labels['app'].value"
            - name: tcp_connections_closed_total
              tags_to_remove:
              - reporter
              - source_workload
              - source_principal
              - source_version
              - source_cluster
              - destination_workload
              - destination_principal
              - destination_version
              - destination_service
              - destination_service_name
              - destination_cluster
              - grpc_response_status
              - connection_security_policy
              - source_app
              - destination_app
              - source_canonical_revision
              - destination_canonical_revision
              - destination_service_namespace
              dimensions:
                destination_canonical_service: "(cluster_name == 'PassthroughCluster') ? upstream.address : upstream_peer.labels['app'].value"

4. Other Pointers

Besides these optimisations we can also focus on reducing cardinality of metrics by reducing the number of possible valuse for a particular label. Mainly, if we record “url” label in the metrics, then we need to reduce its cardinality, either by grouping of urls then recording the group instead of raw url. If you cannot group the list of urls, then atleast we can trim the urls by its length, so that we can achieve less cardinality.

Following EnvoyFilter adds a header, url-category in every requests which has first 15 characters of the original url, thus we can record it as url in our metric system:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: url-category
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function envoy_on_request(request_handle)
                local url_var = request_handle:headers():get(":path")
                local u_cat = string.sub(url_var,1,15)
                request_handle:headers():replace("X-url-category", u_cat)
            end
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function envoy_on_request(request_handle)
                local url_var = request_handle:headers():get(":path")
                local u_cat = string.sub(url_var,1,15)
                request_handle:headers():replace("X-url-category", u_cat)
            end

Thus, if you check now we should have reduced the number of metrics and have eliminated histogram metrics as well, which provides a drastic improvement in the memory consumption of istio sidecar container. Since we have only expose simple metrics like counter in istio-proxy’s /stats/prometheus api, metrics will not grow exponentially as it was for histogram metrics for each requests, metrics will accumulate much slower (but still it accumulates…) until pod restarts.

Takeaways…

As we just discussed, metrics will still accumulate in prometheus stats api of istio-sidecar container, unless we change the method of shipping the metrics. Remember this is not the complete optimization. For further optimization checkout the Part II of this discussion.