The Power Of Kibana in 6 Simple Steps

At Conductor, our systems generate over 10 million lines of logs every day. On a particularly busy day, we can break 30 million lines. When the DevOps team started using the ELK stack (Elasticsearch, Logstash, and Kibana) to help us manage all of that data, I had to learn a whole new skill set.

The techniques that I had always used to search through text logs were no longer applicable. In this blog post, I’m sharing the best of what I learned as I filled my Kibana toolbelt. Now you will be able unlock the power of Kibana, too.

Step 1: Change your view of Kibana log records by using the field selector.

By default, when you enter Kibana, the records you see in the Discover view look something like this:

The message field represents the actual text from the log file, but is accompanied by a lot of metadata that makes it more difficult to scan the page and find what you’re looking for. By adding message to the Selected Fields, you can simplify your view of the data.

Now your data will look a lot more like log output.

Step 2: Respect the rules of the Kibana search tool and Lucene Query Syntax.

Unfortunately, because there were so many records of users hitting the login page and downloading images, the output was still confusing to view. My instinct was to get rid of these by filtering the data using the query “ -login -png”, but that didn’t get me anywhere. I had to use wildcards to fully match the terms I wanted to exclude. “ -login* AND -*png” did the trick.

Partial matches are not displayed and a period (present in both “login.html” and various image names) isn’t treated as a term delimeter. Kibana uses Lucene Query Syntax on tokens created by Elasticsearch , so queries are interpreted differently than you might initially expect.

Even after applying this filter, the data was still noisy because messages from multiple logs were visible. I narrowed Kibana’s search to a specific field, “ source” , to focus on a particular type of log. By entering “ source:*access.log “ in the query bar, I limited the results I could see to messages from access logs.

Step 3: Get your text file using the developer console.

Now I was viewing lines of text taken directly from the access logs of our servers. One final trick allowed me to generate a CSV from that data-a file I could filter and transform however I wanted.

If you enter the following script into your developer console while in the Discover view, you will find a CSV with the visible records copied to your clipboard.

var headers = $("span.table-header-name").toArray().map(h => $(h).text().trim()); var data = $(".kbn-table tr.discover-table-row").toArray() .map(tr => $(tr).children(".discover-table-datafield").toArray() .map(td => $(td).text())); var csvString = [headers, ...data].map(r => r.map(c => `"${c}"`).join(";")).join("\n"); copy(csvString);

(The maximum number of records that you can view, and therefore export, is configurable through the “ discover:sampleSize” setting in the Management / Advanced Settings section.)

Step 4: Explore new fields in the Management section of Kibana.

Although everything I had learned up until this point had made Kibana much easier to use, I knew that there was still more to discover. My teammate had made a chart tracking “ backend_processing_time” for a particular kind of request, and I had yet to encounter any records with that datapoint.

I headed to the Management tab and clicked on Index Patterns. There, I found a list of all of the fields available in the logs that Elasticache had indexed. “ backend_processing_time” was right there. I used the query “ _exists_:backend_processing_time” to find records where it was present.

Step 5: Use Logstash’s Grok filter plugin to extract the fields you need.

I wanted to better understand how “ backend_processing_time” varied across dimensions like which client was logged into the application and what feature they were using, but none of the fields I was interested in aggregating by were available in Kibana. I could determine the relevant information by examining request paths, but I wanted to make that data a first class part of our Kibana output so that it could easily be used in any analysis.

With a little help from our DevOps team, I modified our Logstash parsing configuration to extract the fields I wanted. I used Grok Constructor to build a regular expression for the Grok filter plugin that would generate more useful data from our URLs. I even added some custom patterns . I also used the Mutate filter plugin to make sure that the new fields I created only appeared when they were relevant. Now each log record in Kibana had new Searchlight-specific attributes and I was able to do some interesting analysis.

Step 6: Identify patterns using Kibana visualizations.

To take advantage of one of Kibana’s most powerful tools, head over to the Visualize section using the left nav. I wanted to make a chart, so I clicked on “New” in the top nav to create a new visualization and selected “Vertical bar chart” as my visualization type. I chose the indices I wanted to get my data from. In my case, it was all indices matching the “ logstash-*” pattern. I built my aggregation as illustrated below, and clicked the play button.

Now I had some interesting data visualizations that I could dig into. Kibana is an important tool for our team, and no longer unfamiliar territory. Use this guide to help you make Kibana work for you, help you answer questions, and access and use your log data as efficiently as possible.

Note: This post is based on my experience using Version 5.2.0 of Kibana. Depending on which version you’re using, your mileage may vary.