Log analysis is the process of interpreting computer-generated records, it gives visibility into the performance and health of IT infrastructure.
Pain points of Traditional log analysis:
There are many challenges in traditional log analysis. First is inconsistency, log data structures vary with every source that generates this information. Another difficulty in log analysis is the volume of data. Machine data volume grows significantly with the number of devices, making it difficult to find a solution for an issue since it requires the ability to filter out only relevant information.
Benefits of using Splunk to do log analysis:
Splunk is a centralized logs analysis tool, it makes log analysis much easier with its great features. It can collect data from the local computer as well as from remote network sources that have the Splunk client installed. Splunk can index many types of machine data, and also allows the user to create custom queries based on the data format to find relevant results, correlate them to other data, and present the information in a variety of charts, graphs, and geographic maps.
Use case of analyzing log:
Below diagram shows using Splunk to analyze a user authentication log file in storage. What I used to do is opening the log file and start searching with Linux commands such as grep and regular expression. With Splunk, I just need to import the log to the Splunk server, then all the events are automatically indexed. By clicking these indexes, we can show the events in a categorized manner. Regarding the search function, we can also choose a smaller time range to speed up the search and narrow the search results. If I want to search for domain-related events, I can input ‘domain’ into the search field and a list of events is shown. The bar chart at the top shows the event rate in the time frame in a user-friendly way. This intuitively tells the user what time the events occurred the most.
Then we can use special search commands to transform search results into statistical data tables and create visualizations. Moreover, the search report can be saved into a dashboard for later demonstration.
Since I only have one log file, Splunk’s capability is not fully shown. If we have a set of logs, the powerful search allows us to we can cross-check multiple logs and find the correlation between incidents. One scenario is to use the visualization tool to find the relation between the server’s performance drop and the number of client operations.
