Palantir
Palantir
Dec 13, 2018 · 8 min read

In osquery Across the Enterprise, we report a high-level overview of deploying osquery in a large environment, but touched only briefly on implementation details. This blog post is the first in a two-part series where we will cover osquery’s auditing features in depth. This post lays the foundation by expanding on the basic concepts of the Linux Audit Framework. Part two focusses on concrete osquery configuration and implementation steps.

One of osquery’s most powerful features is its ability to record process executions and network connections. In osquery terminology, these features are commonly referred to as process and socket auditing. However, we’ve observed that many people attempt to enable these features without fully understanding the underlying mechanisms and performance implications. Without a deeper understanding of how osquery’s auditing functionality operates, users may encounter performance issues, resource conflicts, and troubleshooting woes. Additionally, improperly configured auditing can result in incomplete data, which can make detection and alerting programs less effective.

This post will describe how auditing works under the hood and provide pragmatic guidance on how to implement and fine tune auditing configurations.

Terminology

To begin, we’ll clearly define the terminology that we use throughout this post. When we talk about “auditing,” we are referring to a program that processes (and usually logs) audit events that originate from the audit subsystem within the kernel.

To summarize the terminology that we’ll use throughout these posts:

  • Audit Framework/Subsystem: The audit component within the Linux kernel that generates audit messages based on a configurable rule set.
  • Auditing: The process of using the audit subsystem to generate audit events for auditing purposes
  • Audit Consumer: A userland application that processes audit events emitted from the kernel such as auditd or osquery
  • Auditd: The userland daemon responsible for collecting audit events and logging them (usually to the filesystem)

With regards to auditing, both osquery and auditd serve a nearly identical role. Both daemons process incoming audit events generated by the audit subsystem and generate output that is intended to be logged or processed in some way.

Before we begin talking about osquery, we’ll explain how the audit subsystem functions.

Understanding audit

The auditing functionality that exists in the Linux kernel today was introduced around version 2.6 of the Linux kernel, with the primary goal of allowing better introspection into security relevant operating system events such as file modifications and system calls.

Auditing is configured and managed through a set of rules that are stored on the filesystem (usually in a subdirectory of /etc/audit). There are 3 different types of audit rules:

  • Control rules: allow the audit system’s behavior and some of its configuration to be modified.
  • File system rules: also known as file watches, allow the auditing of access to a particular file or a directory.
  • System call rules: allow logging of system calls that any specified program makes.

This post is only concerned with control and system call (syscall) rules. The listing below shows an example of the audit.rules file after it has been updated by osquery:

# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl
# First rule - delete all
-D
# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 1024
# Feel free to add below this line. See auditctl man page
-a always,exit -S execve
-a always,exit -S bind
-a always,exit -S connect

The first two rules are control rules. They modify the core functionality of audit by deleting any pre-existing rules and by setting the buffer backlog size. The next three rules are syscall rules that tell the audit subsystem to always monitor this syscall, and to do so when the syscall exits. The syntax for system call rules is:

auditctl -a action,filter -S system_call -F field=value -k key_name`

A system call is typically how a program in userspace communicates with the operating system kernel. The syscall rules you see in the image above are the rules loaded by osquery when both process and socket auditing are enabled.
To enable process events, the execve() syscall is recorded:

  • execve() — Execute program

To enable socket events, the bind() and connect() syscalls are recorded:

  • bind() — Assigns an address to a socket
  • connect() — Initiates a connection on a socket

It’s important to note that the three syscalls listed above are not comprehensive of all possible process and network activity on a system. Additionally, osquery does not support the processing of all possible syscalls. For example, process forking and cloning will not be recorded by osquery as those operations use the fork() and clone() syscalls and cannot be natively parsed by osquery. Additionally, as outlined in osquery issue #5084, incoming network connections and connections on non-blocking sockets are also not supported at this time.

To determine which syscalls a particular program makes, you can use strace:

# strace curl google.com 2>&1 | egrep 'execve|bind|connect'
execve("/bin/curl", ["curl", "google.com"], [/* 22 vars */]) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("216.58.194.174")}, 16) = -1 EINPROGRESS (Operation now in progress)

Configuring audit parameters with auditctl

To change the core functionality of audit from the command line, you can use auditctl.

root@localhost# auditctl -s
enabled 1
failure 0
pid 2021
rate_limit 0
backlog_limit 1024
lost 0
backlog 0

Auditctl allows you to configure:

  • Whether auditing is enabled
  • If the system should fail silently, verbosely, or panic
  • Rate limit — any number of events exceeding this limit in a single second will not be logged and the failure flag will be set
  • Backlog limit — The queue size of the socket buffer for audit events. Any events exceeding the backlog limit will be dropped and the failure flag will be set
  • Lost: how many events were discarded due to overflowing backlog

Rate and backlog limits are important to understand because they control the behavior of how audit will drop events. In most cases, audit should not be considered a lossless solution. As audit events are generated, they will be stored in a buffer and evaluated against the rate and backlog limits. The backlog limit is effectively the number of events that the buffer will hold at any given point in time, and auditd/osquery pulls events out of that buffer. If the audit consumer can’t empty the buffer as fast as it fills up, audit will simply drop any events that cannot fit in the buffer. The number of dropped events will be displayed in the “lost” line. Additionally, the rate limit sets the number of events allowed per second. Any events above that threshold will be dropped. Both the rate and backlog limits will set the failure flag to “1” if they are exceeded.

Basic audit architecture

Perhaps the single most helpful visual resource in understanding audit is a diagram from Slack’s Syscall Auditing at Scale blog post:

Image credit: Syscall Auditing at Scale (Slack Engineering Blog)

In this diagram, you can imagine the “go-audit” node being interchangeable with both auditd and osquery. When software runs, it (typically) generates syscalls. When audit is enabled, these syscalls are evaluated by the kernel and compared against any audit rules that have been loaded. If an audit event is to be generated based on the configured rule set, the event is then sent to the audit netlink socket, where a userland process such as auditd, osquery, or go-audit listens for and consumes audit events.

osquery documentation specifically mentions not running more than one audit consumer at a time:

auditd should not be running when using osquery’s process auditing, as it will conflict with osqueryd over access to the audit netlink socket.

It appears that kernel versions 3.16+ support multicast netlink sockets, but we are currently unfamiliar with the details required to support this configuration when using osquery. Multicast netlink sockets should theoretically allow users to run multiple audit consumers concurrently without encountering the conflict mentioned in the documentation.

Auditing with osquery

There are a few advantages to using osquery for auditing purposes as opposed to other audit consumers and security tooling. Primarily:

  • No kernel module is required
  • Output filtering via custom queries
  • JSON formatted logs

Some security products that advertise the ability to record network and process information require the installation of a kernel module, the source code of which is usually proprietary. Although kernel modules may improve performance when compared to audit, they sometimes have compatibility issues with different Linux distributions or kernel versions.

With osquery, you can choose to either record all process and/or network events using a wildcard query (SELECT * FROM process_events) or you can use the underlying SQL engine to create a whitelist or blacklist of events that you are interested in logging (e.g., SELECT * FROM process_events WHERE path!="/usr/bin/sed"). Filtering events in this way doesn't reduce the performance overhead in any way, but it can help cut down on the volume of logs being written by osquery. The reason this filtering method does not reduce performance overhead is that osquery has already done the heavy lifting of parsing the incoming audit event and loading it into a virtual table. In part two of this blog post, we'll provide additional filtering methods that also reduce computational overhead.

Lastly, the comparison between the logs generated by auditd and osquery is best displayed using samples from their respective log files. These are the corresponding logs generated by auditd and osquery respectively when running “curl google.com” on a host:

auditd

type=SYSCALL msg=audit(1521667806.157:217090): arch=c000003e syscall=59 success=yes exit=0 a0=fbb6c0 a1=fbb760 a2=fb55c0 a3=7ffe4ead6520 items=2 ppid=2000 pid=42971 auid=890466808 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=26 comm="curl" exe="/usr/bin/curl" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=EXECVE msg=audit(1521667806.157:217090): argc=2 a0="curl" a1="google.com"
type=CWD msg=audit(1521667806.157:217090): cwd="/etc/audit/rules.d"
type=PATH msg=audit(1521667806.157:217090): item=0 name="/bin/curl" inode=306502 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1521667806.157:217090): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=8485228 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:ld_so_t:s0 objtype=NORMAL
type=PROCTITLE msg=audit(1521667806.157:217090): proctitle=6375726C00676F6F676C652E636F6D
type=SYSCALL msg=audit(1521667806.163:217091): arch=c000003e syscall=42 success=no exit=-2 a0=3 a1=7f1c7f451610 a2=6e a3=7f1c7f451b20 items=1 ppid=2000 pid=42971 auid=890466808 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=26 comm="curl" exe="/usr/bin/curl" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=SOCKADDR msg=audit(1521667806.163:217091): saddr=01002F7661722F72756E2F6E7363642F736F636B657400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

osqueryi

osqueryd

In our experience, working with the clean and customizable JSON output format provided by osquery is far better than working with the output supplied by other auditing solutions. Adding or removing columns from the output is as simple as making a one line modification to the SQL query against the process or socket events table.

However, no solution is perfect and there are certain caveats to keep in mind when considering osquery as an audit consumer:

  • Shell built-ins (echo, env, export, etc.) do not generate process events
  • As previously mentioned, audit is not a lossless solution and osquery may also drop events if an event table overflows
  • osquery does not record events in real-time (You can configure an interval from 1s or higher, so it’s close!)
  • Enabling audit comes with a degree of performance overhead, as does running an audit consumer such as osquery. The amount of overhead depends on a few different factors that will be covered in part two of this blog post series.

Conclusion

We hope this post helped you understand how the audit framework works and what configuration options are available to modify its behavior. Part two of this series provides a detailed guide on how to enable auditing for osquery on Linux.


Authors

  • Chris L.

Palantir Blog

Palantir Blog

Thanks to Chris Long

Palantir

Written by

Palantir

Palantir Blog

Palantir Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade