Why Should We Care About Logs in Cybersecurity?
What Are Logs?
Logs, in essence, are files that document specific events, time of occurrence, location of occurrence, and other information related to the incident, depending on the parameters configured. These are absolutely crucial when it comes to security and troubleshooting- whether we are looking at past logs to determine the root cause of a breach or more recent logs to determine whether operations are functioning normally, logs inform us of the events occurring in our system and are pivotal in keeping us informed.
Log Format
Logs can come in various formats. If your logs come from the Windows Event Viewer, you can utilize the .csv, .XML, .evtx and .txt extensions when saving them.
On the other hand, if you are using a Linux system, logs will typically have the .log extension.
Where can I find my logs?
In a Windows environment, event logs will usually be stored in the C:\Windows\system32\config folder. Note that to modify, view, or edit the log files, you will need elevated privileges.
In a Linux environment, logs will usually be stored in the /var/log path. The same logic applies here- depending on the log file, you will need the appropriate permissions to read and/or modify it.
Where does logging occur?
In enterprise environments, almost all devices within the network can be logged. From workstations, endpoints, routers, switches, firewalls, and proxies to IoTs, logging can be a very effective tool in understanding what is exactly occurring in these systems. However, it is important to highlight the importance of quality logging.
Logging events that are irrelevant to your objectives can result in noise, which is often more detrimental than beneficial. You certainly don’t want to go through many files of irrelevant information just to find something that is helpful to you. It can also be very costly in terms of disk storage, which we will discuss next.
The goal should always be to log important events- occurrences that indicate whether operations are functioning as intended, unusual traffic, and so forth.
How much disk storage is used when logging?
Depending on the organization and the devices employed, the number of EPS (Events Per Second) can vary. For instance, on average, a Windows Workstation can have an EPS of 1.0, which denotes one event per second, while firewalls can have an EPS of 10.0–50.0, which denotes 10–50 events per second.
However, it is important to consider peak times when making these decisions as well, as it is often in those times that something unusual is occurring. To find how much space you will be utilizing on a given day, you can use the following equations:
EPS(per device)* 86,400 (number of seconds in a day) * 600 (size of an average raw single event, in bytes) = size of the uncompressed log file in bytes for a given day and a single device considering that events are stored in its raw form.
EPS(per device)* 86,400 (number of seconds in a day) * 1500 (size of an average normalized single event, in bytes) = size of the uncompressed log file in bytes for a given day and for a single device considering that events are stored in its normalized form.
So, you may be asking yourself, what is the difference between a raw event and a normalized event?
A normalized event, in its essence, is a modified version of the raw event. The contents of the raw event are re-organized and extra information is added so that the event becomes more readable and understandable to the security administrator. This is a very important step to ensure that logs are providing the information required to properly secure systems. However, normalized events are heavier in terms of bytes compared to raw events.
Compression
Storing these log files in uncompressed form can take a lot of storage space. Therefore, we must utilize compression to reduce the amount of storage disk requirement. An 8:1 ratio can be a good starting point, but it is important to factor in variables like impact on the SIEM system, amount of normalized data versus raw data, and so forth.
Summary
Logs are a critical element of Cybersecurity operations. When properly executed, logging can provide us with a clear view of what is exactly happening or has happened in our systems, allowing us to corroborate events with more precision and make adjustments as needed. Understanding where and what to log is crucial if we are to extract value from this process. Logging can be costly in terms of storage disk required, so it is important to consider various elements (EPS, NEs (Normal Events), PEs (Peak Events), normalized and raw events, compression rate, number of devices, potential headroom, room for growth, etc.) when allocating disk space resources to log files.
