ElasticSearch Query Audit Kibana Dashboard
Using X-pack Security Audit feature
Are you curious how your users use your Elasticsearch cluster? Have you wondered what types of queries are being submitted by your users?
As I’m taking care of my Elasticsearch cluster, I’ve often run into these questions. I’ve tried few options mentioned by Elasticsearch using packet beat or using slow query log. However, until 6.5 documentation mentioned about query auditing using security auditing features, I didn’t realize that we could gather meaningful insights with very minimal set up.
By the way, packet beat option didn’t work out for my use case because my cluster already have TLS set up. Also, slow query log is very verbose. Once you enable this for an index, every data node where its shards belong to capture query performance information. So I had to fall back to the security auditing option.
I’ll first walk you through 4 steps to create some slick visualizations in Kibana, and wrap up with discussion points.
Simple 4 Steps
1. X-Pack-Related Settings
It is essential to have X-pack (with adequate licenses; at the time of writting you need at least Gold, but you can get 30 days trial version) to enable this. We need to have the following settings:
- If you already have set up security auditing, make sure your
authentication_success. The idea is to visualize subset of
authentication_successevents by filtering requests with queries.
- My Elasticsearch cluster is pretty big so it has a coordinating node designated for search requests and Kibana also points to it. Therefore, it was fair assumption for me to look at request body only at this node. So the setting
xpack.security.audit.index.events.emit_request_bodywas only added to the coordinating node.
2. Override Default Index Template
By default, the index template created for the security audit system index explicitly disabled indexing for
request_body field. You can check it yourself by running
GET _template/security_audit_log on Kibana’s Dev Tools. In order to override the index, run the following command on Kibana.
- Note that it needs to have higher order (2000) than the default one (1000) so that mapping changes in the custom template takes precedence over the default one.
- By making it into text field, it enables query capabilities on the request body.
- By adding keyword field, you can query whether the field is empty or not.
3. Create Kibana Saved Search with Filters
First, create a Kibana index pattern for
.security_audit_log*. Then in discover tab, create a saved search with custom filters. If you don’t know how to create a custom filter take a look at this Kibana guide.
The key idea is to view security events only with request body with queries. (Since security audit log doesn’t just capture search requests) You may need to adjust the filters based on your environment, but you get the idea right?
4. Create Kibana Dashboard
Few tips on how I created the dashboard above:
- Query Count by User: With line chart, used Date Histogram aggregation on X-axis, and also split series by
- Query Profile: With horizontal bar chart, created multiple filter buckets by using various match queries on
- Security audit log data is center around authentication and authorization events. How can we join query performance data? Does it need to be done externally?
- What’s Elasticsearch’s road map on query auditing?
- Should we ship the audit log to monitoring cluster?
Please share your experience and thoughts below!