Integrating Kiali and Grafana Tempo

Updated to reflect the latest changes in Kiali 1.78

Kiali, serving as an Istio console, offers a means to visually represent Istio’s data and configuration. When Istio’s distributed tracing is activated, Kiali has the capability to present trace and span data using a bubble chart.

Bubble chart in Kiali showing Traces for a Workload

This optional feature historically integrates with the Jaeger distributed tracing tool. But with the release of Kiali version 1.75, Grafana Tempo is now also supported as a distributed tracing back-end! The full set of features is available from Kiali version 1.78.

Kiali can both read and send traces. In its default configuration, Kiali reads the traces sent by Istio, from the Jaeger or Tempo API. Additionally, Kiali can be configured to send its own traces to the Trace ingester, for debugging purposes.

Kiali, Istio and Grafana Tempo integration

Tempo has the capability to ingest traces in various formats, such as Jaeger or Zipkin. Kiali sends traces to Tempo in the OpenTelemetry format.

In order to display tracing data from Tempo, Kiali reads the traces from the Tempo API in OpenTelemetry format and internally transforms them into Jaeger traces.

How to configure Tempo

It is quite easy to configure Kiali with Grafana Tempo, just the following configuration is required (assuming the typical ports):

external_services:
  ...
  tracing:
    enabled: true
    provider: "tempo"
    in_cluster_url: http://frontend-query.tempo:3200
    url: http://my-tempo-host:3200 
    use_grpc: true
    grpc_port: 9095
  ...
  grafana:
    in_cluster_url: http://grafana.istio-system:3000
    url: http://external-grafana-url

Kiali links every trace and span with the proper URL for redirection to the Tracing tool. Because Grafana is the default Tempo UI, the Grafana URL needs to be set up also in the Kiali configuration.

Kiali redirects to the trace tool

For Kiali to appropriately redirect to the correct URLs, it’s necessary to establish a default Tempo Datasource in Grafana.

Istio trace in Grafana Tempo Datasource

gRPC is available for accessing the Tempo API. But it will still require the HTTP endpoint to be accessible.

The full set of features have been released. We would love to hear your feedback to keep improving the Grafana Tempo and Kiali integration!