osquery for Security — Part 2
Advanced osquery functionality. File integrity monitoring, process auditing, and more.
osquery for Security was aimed at people who are new to osquery, but this post will dive deeper into some of the more advanced and less known features that osquery has to offer. In addition to scheduling queries against aspects of your operating system, osquery has built in functionality to monitor file integrity, audit network connections and processes, and even log hardware device changes in near-realtime.
Define a Data Collection Strategy
osquery is most useful when you’ve planned out what data you want to be collecting ahead of time.
As tempting as it can be to run tools in stock configurations, much more value will be derived from doing some homework around the data you collect and potential alerts that can be built around that data. osquery doesn’t function like a traditional HIDS/IPS in the sense that those types of products generally fire “alerts” when something looks suspicious. There is a no such thing as a “one size fits all” osquery configuration — you need to see for yourself what data is available to query and make your own decisions about what to collect and how to respond to it.
Also consider how you plan on generating alerts based off of the data you’re collecting. You could write a “bash_reverse_shell” query and generate an alert anytime that event appears, or you could just collect all process information and write the reverse shell detection logic in Splunk/ELK/Graylog and do the detection and alerting logic there. There’s tradeoffs to be made either way, but just make sure you’re being consistent with your collection and alerting methodology.
File Integrity Monitoring
File integrity monitoring in osquery allows you to define a set of system paths or filenames to monitor for creation/modification/deletion.
FIM is enabled simply by declaring a “file_paths” stanza. The docs are pretty clear about how this should work and how wildcards can be used.
In the FIM log sample below, there’s a file with action CREATED and has a path of /private/tmp/hello. Be sure not to confuse the bottom “action” field with the “columns.action” field. The “action” field pertains to the entire event being added/removed from a table and the latter pertains to the FIM event.
If you wanted to extend this functionality even further, osquery also supports file access monitoring. In addition to generating events from file creation/modification/deletion, access monitoring will generate events when a process even attempts to access a file. However, be careful about configuring your list of directories to monitor — certain directories are bound to have files that are accessed often and can negatively impact system performance and fill up logs with noisy events.
All of the events generated from FIM and access monitoring will end up in the file_events table. Simply schedule a query against the file_events table in your conf file to gather the changes that have occurred since the last query interval.
NOTE: By default, osquery holds a maximum of 1000 events per type before overwriting/expiring them. This limit can be increased with the --max-events flag.
Process and Socket Auditing
Full network and process monitoring on endpoints is becoming increasingly popular and many new tools are being released that include this functionality. Here’s my perspective on using osquery for auditing:
Pros:
• Auditd is not required on the host system
• Filtering out events is extremely easy (done via SQL in the config)
• No need to write an auditd configuration file (although they are supported)
• Output logs are much more human readable than the logs generated by auditd
Caveats:
• Events are not retrieved in realtime, they have to be queried from a table on a specified interval
• To my current knowledge, osquery doesn’t currently support tracking users who switch identities (i.e. auditd’s auid field) Edit (4/6/17): This is no longer true: https://github.com/facebook/osquery/pull/3120/files
• Osquery does not log raw syscalls
Process and socket auditing can be enabled by setting the following flags in your osquery.flags file (don’t include the comments):
# Disable receiving events from the audit subsystem
--disable_audit=false# Allow the audit publisher to change auditing configuration
--allow_audit_config=true# Attempt to retain control of audit
--audit_persist=true# Allow the audit publisher to install socket-related rules
--audit_allow_sockets
I recommend setting a short query interval (~10s) when testing. Process events will be sent to the process_events table and socket events to the socket_events table.
NOTE: If you’re considering deploying this in production, be aware that it’s not uncommon for passwords, access tokens, and other secrets to end up in command line arguments. Be aware of this and optionally filter out these events in your query against the process_events table with a query like:
SELECT * FROM process_events WHERE cmdline NOT LIKE %password% AND cmdline NOT like %SECRET%;
If you’d like to test out process auditing, I wrote a script that will bootstrap a testbed with osquery and Go-Audit.
Hardware Events
If you happened to read Waymo’s post about the potential IP theft case, you may have caught this paragraph:
Once inside, he downloaded 9.7 GB of Waymo’s highly confidential files and trade secrets, including blueprints, design files and testing documentation. Then he connected an external drive to the laptop. Mr. Levandowski then wiped and reformatted the laptop in an attempt to erase forensic fingerprints.
If your organization could also benefit from knowing when USB devices are plugged into workstations, osquery can do that too!
Simply schedule a query against the hardware_events table and you’ll see all sorts of information about USB devices being plugged in and unplugged:
OSX Attacks Pack
I have to admit, I’m not a big fan of most of the packs included with osquery because they contradict my philosophy of mapping out your data collection strategy. The last time I checked, different packs had overlapping queries and collected data in bulk.
With that being said, the OSX Attacks pack is great. It’s filled with over a dozen detection methods for known Mac malware. This pack is the closest osquery gets to working like an actual HIDS. I’ve noticed it usually gets updated within 24 hours of new OSX malware or attack methods being published and it can be a very effective way to spot commodity malware on OSX systems. This is also a great opportunity to use snapshot queries — you’ll continue receiving events from infected workstations until they’ve been cleaned up. If you decide not to use snapshot queries, you can filter by the action field to see when a system was infected and cleaned up.
System infection events:
index=osquery name=packs_osx_attacks action=addedSystem cleanup events:
index=osquery name=packs_osx_attacks action=removed
Kolide
If you’re still hungry for more osquery features, I recommend checking out kolide.co. They’ve built a great product and their quickstart Docker-based setup is a breeze. Kolide leverages osquery’s TLS endpoint and distributed query functionality to give you centralized control of your osquery configurations and allows you to run ad-hoc queries.
