In Log, We Trust

A picture is worth a thousand words.

The might of this idiom can be truly appreciated when we look at the picture above. The above picture was grabbed from a linkedin post by the Cyber Security Community on the networking site. I unfortunately do not have any other source to attribute to and have no intent to pose it as my work.

But coming back to what the picture signifies is that the fancy technologies come and go, “event logs” remain the most important source of truth for an incident responder. There have been some interesting discussions in intellectual circles about the difference between an ‘event’ and a ‘log’ where there is a section that feels that every log is an event and there is a section that feels otherwise.

In my opinion too, there is a subtle difference between the two and hence its best to view and treat them with a little difference. I consider every activity that can cause a state change of the system to be an event. Some events are successful and others are not, however they are all events. And the application that was the agent for the event to happen (operating system, front end application, server application etc) may decide to record the event in a pre-determined format and location (syslog/event log/files etc) there by generating a log.

For the incident responder, therefore the collection of events as well as logs is the source of ultimate truth to determining the cause of the incident and extent of damage. The following picture from the SANS describe various types of logs worth monitoring for InfoSec.

But its easier said than done. Aggregating, parsing and searching thru these events as well as logs can be overwhelmingly complex, given their different natures and formats.

When we looked at the osquery project, which aims at simplifying the endpoint interrogation by creating a uniform SQL interface across all the endpoint properties, we challenged ourselves with the idea that if the same interface could be extended to collect, aggregate and parse events as well as logs.

In one of the earlier posts, I demonstrated how the real-time event streaming could be done with osquery and PolyLogyx extension to detect vulnerabilities and breaches.With our latest release, we added the feature of log monitoring and collection with the same form factor of osquery.

At the guts of osquery, every system property needs to be exported as a SQL table. So we created a table called ‘win_logger_events’ where we treat the log recording action of an application as an event itself, thereby combining the philosophies of events and logs.

This table can be configured via the standard osquery.conf file to tell our agent which log files the analyst is interested in monitoring using the following syntax.

“win_logger_plugin”: {

“plugins”:

[

{

“logger_name”: “tail”,

“logger_watch_files”:

[

{

“watch_file_path”: “C:\\temp\\tail.txt”

},

{

“watch_file_path”: “C:\\temp\\tail2.txt”,

“file_regex_pattern” : [“(.*) (\\d+): \\[([^\\]]+)\\] (.*)”, “((.|\\r\\n)*)(secret2)(.*)”]

}

]

}

]

}

Where plugins is an array of different type of log parsers. Currently only text logs are supported and therefore we call the log parser as ‘tail’ because it kind of mimics the unix tail functionality. This name can not be changed when monitoring text based log files. The logger_watch_files is an array of full file paths that need to be monitored, with an optional array of regex patterns to be matched against each log entry. If no pattern is provided, all the log entries are captured in the win_logger_events table, or else only those entries that matched the particular pattern. So in the above example config, we are monitoring 2 files in the folder ‘c:\temp’, one called ‘tail.txt’ for every log entry and other called ‘tail2.txt’ that is monitored for long entries that contain the string ‘secret’.

Now let’s run a small batch file to simulate logging to these files.

C:\temp>type test.bat

setlocal enableextensions enabledelayedexpansion

set /a count = 1

:loop

echo hellotail !count! >> tail.txt

timeout 5

REM sleep 10

echo|set /p=”secret2">>tail2.txt

timeout 5

set /a count += 1

goto loop

C:\temp>

The above batch file will write a string ‘hellotail n’, where ’n’ is a linearly increasing count, in the file tail.txt and a string ‘secret2’ in the file tail2.txt

Now let’s query our osquery table and see what it shows:

osquery> select * from win_logger_events;

+ — — — — — — -+ — — — — — — — — — -+ — — — — — — — — +

| logger_name | logger_watch_file | log_entry |

+ — — — — — — -+ — — — — — — — — — -+ — — — — — — — — +

| tail | C:\temp\tail.txt | hellotail 5|

| tail | C:\temp\tail2.txt | secret2 |

| tail | C:\temp\tail.txt | hellotail 4|

| tail | C:\temp\tail2.txt | secret2 |

| tail | C:\temp\tail.txt | hellotail 2|

| tail | C:\temp\tail.txt | hellotail 3|

| tail | C:\temp\tail2.txt | secret2 |

| tail | C:\temp\tail.txt | hellotail 1|

+ — — — — — — -+ — — — — — — — — — -+ — — — — — — — — +

Viola. The log entries from both the files are now recorded into our monitoring table. These entries can now easily be transported thru osquery’s scheduled query mechanism to a centralize aggregation system.

This was a test demo but imagine this log file to be a web server log file for e.g. iis.log. It would be make it very easy to get all ‘GET’ or ‘POST’ strings from the log file and, (hold your breath for this), extremely easy to monitor for SQL injection attempts if you were to add a regex pattern to search any of the strings define here.

And while we are at it, let me also share this report from one of the top-notch MDR firms on the planet describing the increase in attacks on IIS.

So with one agent now you not only get to do live interrogation, you can also do real-time event collection, application log monitoring and response. And guess what, all this with very easy to use and learn SQL. Beat SQL(injection) with SQL, me says [can that actually become a thing?]