Linux Logging 101
Editor’s Note:
Under its bowel, any operating system such as Linux, prints and records a plethora of system and application activities during operation in the form of logs. Ashish will guide you in learning where to find these indispensable logs, what each log represents, as well as how to interpret them, so that you’ll be well prepared to narrow down (and troubleshoot) even the gnarliest issues when the inevitable comes.
Ashish Bansal is Senior Security Engineer with Traveloka’s Security Operations’ Threat Detection and Response division. He’s responsible for enabling the complete security logging, threat detection, and threat hunting capability in Traveloka.
An operating system (OS) provides the platform to manage diverse hardware and software resources. So, extensive logging in OS can provide you with tons of information about your system/computer that can help you track or remediate issues related to troubleshooting, auditing, or threat hunting.
Usually, logging mechanism is present in most software as a default feature. But advanced level logging of the log aggregation services need a manual setup or rule writing. Logging is crucial especially for applications that have to meet & maintain compliance requirements such as the Payment Card Industry Data Security Standards (PCI DSS).
The availability of these logs simplify logging functions from log aggregation/collection to data storage. Logged event includes information to let you answer questions such as those below that will help you in drawing your conclusions.
Example Questions:
Who performed the events? When did the event occur? What did the event perform on? What tool did the event use? What is the status of the event?
Windows Logging:
In Windows, the logging mechanism is the Windows Event Log that displays logs related to application and system messages, including errors, information messages, and warnings. For example, Event ID: 4624 is for successful logins and Event ID: 4625 is for login failure.
You can check these windows logs three different ways; run the ‘Get-WinEvent’ command in the PowerShell, run the ‘eventvwr’ command in CMD, or just search for ‘Event Viewer’ in the Windows search option.
Windows logs can be collected/aggregated either via Wecutil, using SIEM agents, or you can also check out open source tools like Winlogbeat or Kiwi Syslog Free edition.
I think a lot of people are already aware of Windows logging. But Linux logging is equally important. So, I would cover Windows logging on its own in a future blog.
Linux Logging:
Linux and its applications generate numerous types of messages that are documented in various log files. Such logs are usually the starting point for any Linux sysadmin to perform any troubleshooting-related or reporting purposes task such as collecting server telemetry.
Linux utilizes a number of configuration files, directories, programs, commands, and daemons to create, store, and recycle these log messages. Hence, having insights into where these log files are stored in the system and making use of these useful commands can save a lot of effort and time during troubleshooting and investigation.
Syslog daemon is the background logging process in Unix/Linux and it is the general standard for logging system and program messages in the Linux environment. Everything from kernel events to user actions are logged by Linux, enabling you to see/track any server-related activity. Therefore, in the latter part of this section, I’ll cover where to find these logs and how to use them.
Linux System Logs
The log files generated in Unix/Linux can be classified into four main categories:
- Application Logs
- Event Logs
- Service Logs
- System Logs
Linux has a dedicated directory for storing these logs in /var/log.
It includes logs from the OS, services, and other applications enabled on the system. The directory structure is explained below:
/var/log/auth.log in Ubuntu and /var/log/secure in Redhat holds all authentication related events such as successful or failed logins, root user actions, and output from pluggable authentication modules (PAM).
Try sudo grep "Failed\|sshd" /var/log/auth.log | less
PAM verifies user identity and handles authentication for applications and services that require password. So, rather than writing authentication into individual applications, authentication is handled by PAM. Refer to the screenshot below for pam configuration and directory structure.
PAM can be configured via /etc/pam.conf and this determines the authentication services to be used and the order of services to be used. This conf file can be edited to select authentication mechanisms for each system entry application.
PAM Directory: /etc/pam.d/ directory contains the PAM configuration files for each PAM enabled application and services.
/var/log/dmesg contains device driver messages and gets reset on every boot. It may help you troubleshoot hardware or boot up issues. The file can be pretty exhaustive. So you may want to use dmesg | less command to show screenful messages each time.
/var/log/boot.log contains boot up related details and logged startup process messages. This log can help you in investigating improper shutdown or boot up failures related issues. If you are on Ubuntu v15.04 & above, however, you won’t find boot.log anymore.
So, you can either refer to the logs from running dmesg or journalctl (use the -b flag to get information from your most recent boot or the ‘ — list-boots’ flag to compare between older and last/current boots).
systemd is Linux’s system & service manager and you can query its contents with journalctl.
/var/log/kern.log/ contains kernel events, errors, and warning logs.
/var/log/syslog in Ubuntu and /var/log/messages in RHEL are usually encyclopedic log files that maintain all global system’s activity information, excluding auth-related messages and maintains records of almost everything that happens on the system.
More information on Syslog: The daemon that collects and processes the syslog messages is the Syslogd. It also creates a socket in /dev/log to listen to events, can record messages of local files, or even forwards messages to a remote server. (I will cover the Syslog server setup in my future article). So, the same utility can be used for setting up a log collector/aggregator for Linux and other devices.
/var/log/cron maintains logs and status about scheduled tasks (cron jobs) and other crond-related messages. Crond is a daemon used for scheduling scripts and commands. Run crontab -l to check scheduled cron jobs.
In Ubuntu you can get the Cron related logs in var/log/syslog, so just do grep CRON /var/log/syslog
/var/log/apt/ directory in Ubuntu and /var/log/yum.log in RHEL contain logs related to package installation using apt and yum respectively.
Ubuntu contains the following two apt-related log files:
- /var/log/apt/history.log: It contains a list of APT commands, who ran them, and what was installed, upgraded or removed.
2. /var/log/apt/term.log: It logs the information regarding output of apt terminal commands.
/var/log/lastlog contains details about every user’s last login in non-human readable format. But, that database can be queried with the `lastlog` command. You can also print the log file one screen at a time by piping its output to less as in `lastlog | less`
/var/log/wtmp is a binary file that maintains details regarding each login/logout and you can run who or last to get information about current or recent user logins respectively.
Some other commands that use the wtmp database includes `who, w, whoami and last -aiF`
/var/log/btmp maintains records of all failed login attempts and are similar to /var/log/faillogs but you can utilize the lastb command to get the result for all failed logins.If you try to open the btmp.log then it would look similar to below screenshot.
Refer to screenshot below for btmp.
Commands: `lastb` or `lastb -aiF`
Some of the application-level logs can also be found in the /var/log/ directory. For example, Apache web server and MySQL write logs to /var/log/apache2 and /var/log/mysql respectively.
I hope this article has provided a general overview of logging mechanisms in Linux and the utility of these logs.
If you find this topic exciting or interesting, this is just the beginning of the roads ahead as there are multitude of other security and data logging concepts. Come and explore available career opportunities to join us at Traveloka.
