Jaeger Elasticsearch and Kibana
In the latest release of Jaeger 1.7 we have added a couple of major features to improve the user experience and integration with other tools. One of these features is a better support for querying Jaeger data in Kibana.
This feature allows storing tags as object fields, as opposed to an array of nested objects, to overcome the issue. The issue is that Kibana does not support query or aggregations on nested objects. This functionality can be simply enabled by --es.tags-as-fields.all=true.
Let’s look at an example query in Kibana. The last section of this blog post contains instructions on how to run it on your machine.
The query tag.http@url:*customer=123 shows all spans containing a tag http.url with a value matching regex *customer=123. Using the regex is a powerful way to search for chaining patterns in tag values.
Now let’s create a visualization showing numbers of different span kinds. First a filter has to be created with tag.span@kind:*, then the filter is used in the visualization and the chart is aggregated based on different span kinds.
Finally we can create a dashboard with multiple visualizations — aggregations based on customer ID, number of spans per service, span kinds and the total number of error spans.
When the pointer is hovered on a bar it shows the total count and aggregation information — tag.span@kind=server or tag.span@kind=client. From the screenshot we can see that total number of client spans is higher. It means that corresponding server requests are not traced or the requests failed. The other yet related reason can be that request go outside of our system. This approach is also useful extract business relevant information from the tracing data that can be aggregated and presented to users to provide valuable insights into their business.
To summarize the better Kibana support allows us to:
- Leverage Lucene query capabilities e.g. wildcards which are not supported by Jaeger UI due to limitations of other storage implementations.
- Create dashboards providing an executive view on the monitored system.
Diving deeper
We have shown a flag which instructs Jaeger’s Elasticsearch storage layer to store all tags as object fields. However this feature can be further customized:
--es.tags-as-fields.allstore all tags except binary type as object fields.--es.tags-as-fields.config-fileconfig file with tag keys which will be stored as object fields.--es.tags-as-fields.dot-replacementdot replacement character. Dots in tag keys have to be replaced because Elasticsearch interprets dots as object fields.
The flag--es.tags-as-fields.config-file is useful when it is expected that the monitored system will report a high number of unique tag keys. Elasticsearch has a limitation on a unique number of field names per index. This number is controlled by index.mapping.total_fields.limit(default 1000). Given Jaeger index schema it means that a standard Jaeger deployment can store ~940 unique number of tag keys. Bear in mind that once this number is reached Jaeger will not be able to store new data if it contains a new tag key.
Try the example
The following commands provide a simple way to deploy Elasticsearch, Kibana and Jaeger as docker containers. The commands also deploy Hot R.O.D. — Rides on Demand example app which is used in this post to generate tracing data.
$ docker run --rm -it --name=elasticsearch -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:5.6.10$ docker run --rm -it --link=elasticsearch --name=kibana -p 5601:5601 docker.elastic.co/kibana/kibana:5.6.10$ docker run --rm -it --link=elasticsearch --name=jaeger -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://elasticsearch:9200 -e ES_TAGS_AS_FIELDS_ALL=true -p 16686:16686 jaegertracing/all-in-one:1.7$ docker run --rm -it --link=jaeger -p 8080-8083:8080-8083 jaegertracing/example-hotrod:1.7 all --jaeger-agent.host-port=jaeger:6831
First open a web browser on http://localhost:8080/ and generate some requests by clicking on blue boxes. The app reports various tracing data to Jaeger which should be stored in Elasticsearch. Now we can open Kibana http://localhost:5601 and configure an index pattern with jaeger-span-*. Now all should be ready and configured to create queries and visualizations. The visualizations can be imported via Management — Saved objects — Import and refresh Kibana.
