Export Istio Metrics to New Relic
Istio is an open source service mesh platform that provides a way to control how microservices share data with one another. Istio’s robust tracing, monitoring, and logging features give you deep insights into your service mesh deployment. Istio generates detailed telemetry for all service communications within a mesh. This empowers operators to troubleshoot, maintain, and optimize the applications without imposing any additional burdens on developers.
The installation of Istio Service mesh comes with a Prometheus addon. This Prometheus instance is pre-configured to collect the metrics generated by all the Istio components in the service mesh. These metrics can be used to build operations dashboards to monitor the health and performance of your application. However Prometheus addon that comes with Istio is not tuned for performance and security, and not intended for production scale monitoring.
For production scale monitoring of large distributed systems, you could either choose to setup a monitoring solution yourself or leverage any cloud based monitoring solutions. Both options have their own advantages and disadvantages. In this article, we will explore various options to export Istio metrics to New Relic, a cloud based observability platform. Collecting Istio metrics in New Relic is particularly beneficial, if you are already using New Relic for Application Performance Monitoring (APM) and Infrastructure Monitoring. You will be able access Istio metrics alongside other metrics in New Relic while troubleshooting issues in your Kubernetes clusters.
New Relic offers an integration for Istio that sends telemetry data to a New Relic account. This New Relic adapter relies on Mixer, an Istio control plane component responsible for providing policy controls and telemetry collection. But, Mixer was deprecated in Istio v1.5 and not recommended for production usage — Mixer has a significant impact on performance as as each service-to-service communication by a sidecar proxy requires connections to Mixer for metrics reporting.
In version 1.5, Istio introduced a re-architected telemetry feature with reduced resource consumption and lower latencies — Telemetry V2. With this new architecture, envoy proxies themselves provide Prometheus metrics endpoints to expose the metrics. The Prometheus addon installed with Istio scrapes the metrics from these endpoints. However, New Relic does not provide an integration for Istio Telemetry V2 (as of August 2020).
We will explore a couple of options to export Istio telemetry data to New Relic.
Option 1: New Relic Prometheus OpenMetrics Integration
OpenMetrics integration collects telemetry data from services that expose metrics in a format compatible with Prometheus. Metrics from services and pod in a Kubernetes cluster are collected without relying on a Prometheus server. This eliminates the overhead of managing storage and availability of the Prometheus server.
OpenMetrics Integration can be installed in a Kubernetes cluster by following the instructions documented in New Relic Docs.
The installation creates a Kubernetes deployment that collects telemetry data by scraping the target endpoints. The installation also includes a config map with configurations for the endpoints to be scrapped and the metrics to be filtered.
As Istio exposes metrics on a non-standard path and port, additional annotations (prometheus.io/port and prometheus.io/path) need to be added to Istio components and Istio based workloads. Without these annotations, the integration cannot discover Istio metrics endpoints.
Adding annotations to Istio Ingress Gateway and Pilot. We will make use of IstioOperator API to add annotations to Istio components. Update (or create) IstioOperator resource with podAnnotations for ingressGateway and pilot as shown below.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istio-operator
spec:
profile: default
components:
pilot:
k8s:
podAnnotations:
prometheus.io/port: "15014"
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
podAnnotations:
prometheus.io/port: "15090"
prometheus.io/scrape: "true"
prometheus.io/path: "/stats/prometheus"
values:
gateways:
istio-egressgateway:
enabled: false
istio-ingressgateway:
sds:
enabled: trueCopy the above contents into a file istio-operator.yaml. Execute the following command to apply the changes.
$ istioctl manifest apply -f istio-operator.yamlAdding annotations to Istio based workloads. The Istio sidecar proxy exposes metrics on port 15090 and path /stats/prometheus. Update Kubernetes deployments to add the following pod annotations under spec.template.metadata.annotations
prometheus.io/port: "15090"
prometheus.io/scrape: "true"
prometheus.io/path: "/stats/prometheus"The integration will now inspect these annotations and use them to construct target endpoints.
The integration will also export metrics from other services (eg. Kubernetes pods and services) that expose metrics in a format compatible with Prometheus. With the help of filter configuration provided by the integration, we will prevent sending metrics from other services. The following filter configuration will ignore all but Istio and Envoy metrics.
ignore_metrics:
- except:
- istio_
- envoy_Add this configuration to nri-prometheus-cfg config map resource defined in nri-prometheus-latest.yaml manifest file and apply the changes.
Option 2: Prometheus remote write integration
Prometheus remote write integration allows you to forward telemetry data from a Prometheus server to New Relic. This can achieved by configuring a remote_write url in Prometheus. In this approach, we will rely on the Prometheus addon installed with Istio to forward the metrics to New Relic.
1.Generate remote_write url from New Relic Prometheus remote write setup page. The name of the Prometheus server provided will be created as an attribute on the data to identify the source prometheus server or cluster. You can even consider providing the name of the cluster here.
2.Update Prometheus configuration with remote_write url, by editing the config map.
$ kubectl edit cm prometheus -n istio systemThis should be added at the same indentation level as the global section. A sample snippet is given below (non-relevant sections are left out).
global:
scrape_interval: 15sremote_write:
- url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key={LICENSE_KEY}&prometheus_server={PROMETHEUS_SERVER_NAME}
3.Restart Prometheus Server.
$ kubectl rollout restart deployment/prometheus -n istio-systemSince Istio Prometheus addon is pre-configured to scrape metrics only from Istio components and Envoy proxies, (unlike Option 1) no additional configuration is required to ignore/filter metrics from other services running inside the cluster.
Summary
- The detailed telemetry generated by Istio empowers operators to troubleshoot, maintain, and optimize the applications without imposing any additional burdens on developers.
- We explored a couple of options to export Istio metrics to New Relic — Prometheus OpenMetrics Integration & Prometheus Remote Write Integration. When troubleshooting issues in the cluster, Istio metrics collected in New Relic can be accessed alongside the metrics gathered by New Relic APM and cluster explorer.
