ElasticSearch Query Audit Kibana Dashboard

Using X-pack Security Audit feature

Mar 2 · 3 min read

Are you curious how your users use your Elasticsearch cluster? Have you wondered what types of queries are being submitted by your users?

As I’m taking care of my Elasticsearch cluster, I’ve often run into these questions. I’ve tried few options mentioned by Elasticsearch using packet beat or using slow query log. However, until 6.5 documentation mentioned about query auditing using security auditing features, I didn’t realize that we could gather meaningful insights with very minimal set up.

By the way, packet beat option didn’t work out for my use case because my cluster already have TLS set up. Also, slow query log is very verbose. Once you enable this for an index, every data node where its shards belong to capture query performance information. So I had to fall back to the security auditing option.

I’ll first walk you through 4 steps to create some slick visualizations in Kibana, and wrap up with discussion points.

Simple 4 Steps

1. X-Pack-Related Settings

It is essential to have X-pack (with adequate licenses; at the time of writting you need at least Gold, but you can get 30 days trial version) to enable this. We need to have the following settings:

  • If you already have set up security auditing, make sure your xpack.security.audit.index.events.exclude doesn’t have authentication_success . The idea is to visualize subset of authentication_successevents by filtering requests with queries.
  • My Elasticsearch cluster is pretty big so it has a coordinating node designated for search requests and Kibana also points to it. Therefore, it was fair assumption for me to look at request body only at this node. So the setting xpack.security.audit.index.events.emit_request_bodywas only added to the coordinating node.

2. Override Default Index Template

By default, the index template created for the security audit system index explicitly disabled indexing for request_body field. You can check it yourself by running GET _template/security_audit_log on Kibana’s Dev Tools. In order to override the index, run the following command on Kibana.

  • Note that it needs to have higher order (2000) than the default one (1000) so that mapping changes in the custom template takes precedence over the default one.
  • By making it into text field, it enables query capabilities on the request body.
  • By adding keyword field, you can query whether the field is empty or not.

3. Create Kibana Saved Search with Filters

First, create a Kibana index pattern for .security_audit_log*. Then in discover tab, create a saved search with custom filters. If you don’t know how to create a custom filter take a look at this Kibana guide.

The key idea is to view security events only with request body with queries. (Since security audit log doesn’t just capture search requests) You may need to adjust the filters based on your environment, but you get the idea right?

Example of Filters in Discover Tab

4. Create Kibana Dashboard

Example Dashboard (Click to Zoom In)

Few tips on how I created the dashboard above:

  • Query Count by User: With line chart, used Date Histogram aggregation on X-axis, and also split series by principal descending.
  • Query Profile: With horizontal bar chart, created multiple filter buckets by using various match queries on request_body.

What’s Next?

  • Security audit log data is center around authentication and authorization events. How can we join query performance data? Does it need to be done externally?
  • What’s Elasticsearch’s road map on query auditing?
  • Should we ship the audit log to monitoring cluster?

Please share your experience and thoughts below!

Daniel Ahn

Written by

Software engineer, learner, listener https://github.com/danielyahn

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade