THM Log Universe

Dehni
Dehni’s Notes
Published in
6 min readNov 7, 2023

THM Log Universe Module Notes

Logs

Logs are the footprints of digital components and invaluable tools and resources in the storytelling of systems, applications and networks.

There are various log sources to deal with, so categorising them is a great practice to start. Data sources can be divided into two main categories.

Note that each data source produces different logs and serves a different purpose.

Multiple data sources generate different logs for different purposes. To avoid log noise and to enable log management, correlation, and investigation, some “minimum qualification requirements” must be present in each log. Common base qualifications are listed below:

  • The system which created the log.
  • The log creation time (date, time and timezone).
  • The event that caused the log to be created (message).
  • The severity of the log.
  • The source associated with the log (IP and port, MAC address, username, or system name).

There are a variety of tools used in log analysis and discovery phases. Each tool has advantages and disadvantages, so the tool preference varies between analysts and the environment. One of the most important details of the log analysis tool is flexibility, compatibility, and support for parsing multiple log formats.

Windows Event Logs

Windows event logs provide in-depth footprint information on the system, security, and applications installed on a Windows operating system. Windows provides a generous amount of logs, and you will need to activate them according to your visibility needs and capacity.

Below are the main specifications of the Windows event logs.

By default, there are two main log categories in Windows. The details are shown in the table below.

The table below highlights the commonly used event logs.

main parts of the log file:

PID and TID data are beneficial for process tracing, correlation, and understanding the natural flow of events during log analysis. The timestamp also plays an essential role in determining which processes and threads were running at a particular time or during the workflow and, if applicable, the call times of the child processes and threads created by the user or execution flow.

Some useful Windows event log IDs are listed below.

The “TargetUserName” field highlights the account name affected by the event. Attackers can create accounts to maintain access to victim systems. They deliberately make subtle spelling mistakes to prevent or delay the detection of the accounts they create by making them look legitimate. Since the difference between a legitimate and a fake name is usually a single letter or symbol, it can be challenging to spot these account services at first glance unless you look closely. MITRE ATT&CK ID: T1136, Detection: DS0002.

Linux Logs

Linux logs provide in-depth footprint information on the system, security, and applications installed. The logging scope is fully configurable, and the default settings are not enough for the current state of the threats.

Common Logs Generated by Unix-like Systems

Cleartext Formatted Logs

Cleartext logs can be viewed and analysed using multiple tools, as highlighted in the above table.

Logs Require Specific Binary to View

Some binary formatted logs can be viewed and analysed only with the corresponding tool.

The Syslog log file structure is defined by RFC 5424–5426. Also, it is widely used and easy to process with native granular system tools like tail and grep. The structure details are explained in the following table.

While RFC defines the main structure, there are alternative implementations of message presentations. The common structure (IETF) is shown below.

  • IETF log format:

timestamp hostname process[pid]: message

Analysing multi-line logs requires high attention to detail and the ability to filter and extract significant details. This process involves identifying and filtering timestamps, hostnames, process IDs, and event descriptions, among other critical details. By effectively analysing and correlating these attributes, it is possible to uncover the sequence of events, diagnose problems, and gain valuable insight into system behaviour, security incidents, or operational performance.

In summary, while individual log entries may seem manageable, the complexity arises when dealing with large log datasets. The magic lies in organising the extraction and analysis of key attributes to reveal the underlying story hidden within the logs, making log management and monitoring an essential part of maintaining system health and security.

Misc Logs: Application Logs

Misc logs provide in-depth footprint information on application-based events, giving more insights on application and process-based details that will help analysts in security operations, including monitoring, threat hunting, and incident response.

Apache logs:

Access logs are invaluable records generated by web servers, containing essential attributes that form the backbone of effective log analysis. These attributes, including IP addresses, timestamps, HTTP methods, URLs, status codes, and user agent information.

These attributes enable administrators and analysts to ensure server health, diagnose problems, detect security threats, and optimise web services for a seamless user experience.

The anatomy of the Apache access.log’s “Combined log format” is detailed in the given table.

Apache access logs are identified by the “Common Log Format” and “Combined Log Format”. Both formats focus on visibility but serve different purposes. The standard format is more lightweight and focuses on the basics, capturing only the essentials such as IP, URL and response. The combined format captures more details, such as referrer and user agent data, which is useful for in-depth analysis. The combined log format structure is shown in the below section.

  • Combined log format

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

Error logs are an essential component of web server management, providing critical insight into system health and potential issues that can impact server performance and user experience.

The anatomy of the Apache error.log is detailed in the given table.

Unlike access.log, error.log is identified under a single format named “ErrorLogFormat” and focuses on providing additional information to the log message. The structure is shown in the below section.

  • ErrorLogFormat:

ErrorLogFormat "[%t] [%l] [pid %P] %F: %E: [client %a] %M"

--

--