Exploring Audit Daemon for Threat Hunting

Recently I’ve been working for a client who wanna enhance their threat detection capability especially for Redhat. Speaking of what are they having at the moment, they are just like most of the organisation which monitoring standard files in linux, like /var/log/messages, /var/log/auth.

And because of their organizational restriction, my hand is tied to suggest them using some well known and handy open source tool like osquery or etc. The only option left, is audit daemon.

I believe no one feels strange to auditd, it is not something new. But somehow applying auditd for threat hunting is never be a common topic on the Internet. My guess would be configuring auditd to benefit your threat hunting process requires deep understanding of different attack techniques in different stages. And auditd rule look complex. I wish these series of “Exploring Audit Daemon for Threat Hunting” could change your impression in applying auditd rule as your threat hunting tool.

Auditd for threat hunting 101

As there is plenty of tutorial on the Internet focusing on how to configure auditd, I’m not intended to repeat the same thing here. If you are new on setting up auditd, I would suggest you coming back after reading the tutorial from Digital Ocean https://www.digitalocean.com/community/tutorials/how-to-use-the-linux-auditing-system-on-centos-7.

The prerequisite of leveraging audit daemon for threat hunting is completely rely on your rules configuration. In your /etc/audit/audit.rule, it is primarily built by below 2 type of rules:

1. File system rules (-w <path_to_file> -p <permissions> -k <key_name>)
2. System call rules (-a -S <system call>)

These 2 type of rules are just the way of expression of your condition, they both generate the same structure of logs from kernel. Below examples may inspire you which situation is better utilise file system rule or system call rule.

File system rules

The reason of using file system rule is when you want to:

• try to capture who executed the binary
• try to capture who altered the config file
• do FIM (File Integrity Management) for some files.

A simple example, I’m gonna apply a single “file system rule” in /etc/audit/audit.rules.

-w /etc/shells -p rwxa -k T1156_bash_profile_and_bashrc

Then trigger the condition by this command.

#echo "#PoC" >> /etc/shells

And generated 4 line of logs in /var/log/audit/audit.log

type=PROCTITLE msg=audit(1550823038.053:47977): proctitle="bash"
type=PATH msg=audit(1550823038.053:47977): item=1 name="/etc/shells" inode=140717 dev=fc:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=SYSCALL msg=audit(1550823038.053:47977): arch=c000003e syscall=2 success=yes exit=3 a0=2077fc8 a1=441 a2=1b6 a3=7ffe6b7245c8 items=2 ppid=13136 pid=13137 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=446 comm="bash" exe="/bin/bash" key="T1156_bash_profile_and_bashrc"
type=PATH msg=audit(1550823038.053:47977): item=0 name="/etc/" inode=130561 dev=fc:00 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT

As you can see, a simple >> write action matched the condition (-p w). The auditd generated 3 type of logs in this event in 4 entries associated with the audit ID (47977) while Multiple records can share the same ID if they were generated as part of the same Audit event.

• type SYSCALL. It tells us the success state of the command, parent process id, process id, uid, gid, command, and our custom key (from auditctl -k).
• type PATH. It tells us the context (/etc/shells). If there is an incident happened, PATH provides you great insight of which file is intact or not.
• type PROCTITLE, it MAY capture the complete command set that being executed. Usually it will be in hex format, require conversion to ascii. It is worth to mention PROCTITLE is not perfect to capture the executed command. In a few narratives like this example, it only captured “bash” instead of “echo ‘#PoC’ >> /etc/shells”.

System call rules

The reason being of using system call rules instead of file system rules is when you want to :

1. try to capture specifically a group of users who attempt to execute binary
2. setup a rule with syscall function context, such as file deletion, change of uid

-a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k T1107_File_Deletion

Then trigger the condition by executing command:

#rm poc.txt

3 line of logs are generated

type=PROCTITLE msg=audit(1550858867.330:48130): proctitle=726D00706F632E747874
type=PATH msg=audit(1550858867.330:48130): item=0 name="/home/bob" inode=394713 dev=fc:00 mode=040755 ouid=1000 ogid=1000 rdev=00:00 nametype=PARENT
type=SYSCALL msg=audit(1550858867.330:48130): arch=c000003e syscall=263 success=no exit=-2 a0=ffffff9c a1=7460d0 a2=0 a3=15e items=1 ppid=13725 pid=13760 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=446 comm="rm" exe="/bin/rm" key="T1107_File_Deletion"

These 3 line of logs are generated without putting file system rule (-w “/bin/rm”).

Instead, “rm poc.txt” matched the condition “-a -S unlinkat”. While you look into the 3rd line of logs, you can see there is a field “syscall” with value “263”. And “263” mean unlinkat function.

Build your own rule, then start hunting

Prepare your own auditd rule is a painful process. Especially when you considering to apply system call rules. If you feel frustrated to build rule from scratch, you may reference to some mature rule on GitHub. I found that bfuzzy’s rules is a very good reference. It also mapped to mitre attack framework, it gives you a very organized rule to learn or even apply in your environment. I believe after you reading this thread, you will at least understand the rules wrote by others. Feel free to check out bfuzzy in Github https://github.com/bfuzzy/auditd-attack.

The next post, I will show you how auditd led us to successful threat hunting in a simple privilege escalation simulation in linux. Please feel free to leave me comments or feedbacks.