What is log analytics?

This Blog has been moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

In a computing context, a log is the automatically produced and time-stamped documentation of events related to a particular event. All systems and applications produce log files.

Log analysis helps us in analyzing log data to derive useful metrics in monitoring, performance, digital marketing, etc. Log analytics helps us in performing real-time analysis of large scale data and obtain insights for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, IoT, etc.

Logs can contain information, alerts, warning or fatal errors. Logs can originate from systems and applications. Logs can be either in a text or in a binary format. Text format is usually preferred by many as it is easy to work with. Binary formats need to be processed first before any analysis can be done. They can also use structured data types such as date and time. To get useful insights from the log files they have to be processed first, independent of where they originate from, what information they contain and their format.

Log data is divided into two

1. Event log (systems and components performance) — provides a thorough view into how our system and their components are performing at any point in time, like whether your servers are running fine, if there are any network failures or any network abnormalities.

Event log

2. User logs (user behaviour) — helps in getting a much better understanding of the online user behaviour such as what they did on the website, things they clicked on during their visit etc.

User access log: User access logs gives information such as from which IP user is making request and status of the request etc. Using this we can know if the user poses a threat to our system.

User request log: here we can see that the user called /image/search API with certain parameters. Using these we can know if the user is using the system as per their job title. This can also help to train the users better in case they are making mistakes. We can also improve the systems if the same mistakes are made by a large number of users.

If implemented correctly in the concerned environment, there are many uses for log analysis. It helps in determining trends. It increases security awareness, detection of protocol failure, failed processes, network outages etc. Using log analytics, businesses can analyse large scale data and obtain useful insights such as:-

1. You can determine the list of URL’s visiting your site or a particular page (popularity) and what the most popular times for visits are, from a web log. You can measure growth over time with this data.
2. Track each individual user to see the sites they visited.
3. Look for patterns that could result in an error and thereby create an alert.
4. Can be used for business analytics by understanding the user. Example: depending on the user’s history suggesting new ads, pages etc.
5. Artificial ignorance: machine learning process that ignores log data that is not useful and detect anomalies. Can also alert about routine events that should have happened but did not.
6. Helps in application troubleshooting by pinpointing areas of poor performance, identifying root cause of application installation and run time errors.
7. Monitor in real time and create specific alerts.
8. In depth analysis of the security logs can give information about attempts to breach security and attacks like virus, trojan etc.
9. Can detect and alert on suspicious behaviour such as when a single user logs in from two different locations at the same time.

Log analysis helps in near real time monitoring and alerting. By analysing log data, organizations can be more aware of potential threats and other issues, find their root causes and mitigate risks much better. All of this is possible only if the data is processed correctly so that the needed information can be extracted and presented, depending on the environment.