Blue Team Diary, Entry #1: Leveraging Osquery For Enhanced Incident Response & Threat Hunting (Free Video Training)
Learn how to use and scale osquery to (proactively) detect evasive malware, stealthy persistence, fileless malware, unseen attacks etc.
Hi there,
This is Dimitrios Bougioukas, Director of IT Security Training Services at eLearnSecurity.
As stated in my first post “Red Team Diary, Entry #1: Making NSA’s PeddleCheap RAT Invisible”, eLearnSecurity has inaugurated a series of posts (diary entries) that will share both IT Security research endeavors and cutting-edge attacking or detection techniques.
My second post will be blue team-oriented and will cover how to leverage and scale osquery for enhanced DFIR and Threat Hunting against modern attacks.
I got familiar with osquery when developing eLearnSecurity’s Incident Handling & Response Professional course and I can assure you it is a really great endpoint monitoring solution.
Note: If you want to cut straight to the chase scroll down. You will find a lengthy and narrated video teaching you osquery fundamentals as well as how to leverage and scale osquery to (proactively) detect evasive malware, stealthy persistence, fileless malware, unseen attacks, etc. You will also learn how to leverage MITRE’s ATT&CK to first emulate and then try to detect unseen attacks with osquery.
At eLearnSecurity we very much like hands-on and realistic training. For this reason every attack mentioned above was simulated in our environment. This means that every osquery-based detection you will see in the video can be directly applied to your environment.
Why osquery?
It is a known fact that every organization should work on maximizing its endpoint visibility. Network traffic visibility is, of course, equally important but modern enterprise networks can pose multiple obstacles to effective traffic capturing and analysis.
As we already know, endpoint visibility can be achieved to a certain degree by deploying Sysmon, but the addition of a specialized agent that can efficiently interrogate endpoints can prove useful during incident response activities. In addition, a specialized agent is great for gaining visibility into Linux endpoints and servers, where Sysmon is not applicable.
Enter osquery…
Osquery is an open source solution that can extend a Blue team’s endpoint visibility in an efficient and centralized manner. It does so by exposing operating system configuration data in the form of relational database tables. All Blue team members have to do is submit (or schedule) queries against these tables to collect valuable data about the current state of an endpoint/server as well as changes performed on it over time.
The great thing about osquery (especially when deployed on a Windows environment) is that it provides detailed insight into the registry, WMI, hardware events and many other areas that were previously disregarded by other endpoint monitoring agents or could not be interrogated though a single endpoint monitoring solution.
That being said, osquery wouldn’t be a great endpoint monitoring solution if it couldn’t easily scale don’t you think?
Imagine being able to execute the same query across all your endpoints and not only that, but also being able to receive an alert in case of a deviation from the normal endpoint state. Well, you can actually do that by enrolling every osquery agent to a Kolide Fleet. Kolide Fleet is essentially an open source osquery manager that enables you to track, manage, and monitor your entire infrastructure from a single user interface
Most of osquery’s virtual tables are generated when an SQL statement requests data. This is not the optimum approach from a data gathering perspective though.
Consider the processes table: if a process like ps runs for a fraction of a moment there’s no way SELECT * FROM processes; will ever include all the details. To solve this osquery exposes a publish-subscribe framework for aggregating operating system information asynchronously at event time, storing related event details in the osquery backing store, and performing a lookup to report stored rows at query time.
Let’s take for example the need for file integrity monitoring (FMI). A query scheduled to run every 5 minutes to identify editing/injection attempts against important files on a system may miss an interval where an attacker tampered with a file. In such cases, where continuous visibility is required, we can leverage the aforementioned publish-subscribe framework. Specifically, we can work with event-based queries and evented tables that track time or action changes to files specified in configuration data. Technical details on how to work with evented tables are provided on the video that follows…
Osquery is full of useful capabilities and integrations. I strongly suggest you go through the official documentation to learn about them. https://osquery.readthedocs.io/en/stable/
It’s training time…
For osquery fundamentals as well as numerous osquery detection examples please view the narrated video below.
The Red Team Perspective
As with any third-party solution deployment or newly-introduced solution, not everything is secure out of the box.
Common deployment shortcomings and some of osquery’s insufficiently secure libraries can be leveraged by red team members to perform stealthy enumeration. Specifically, red team members may be able to perform osquery injections and leverage deployment misconfigurations in order to perform stealthy enumeration or even detailed reconnaissance.
Consider the below web application source code that exposes osquery functionality leveraging an osquery library. By closely looking at the source code, it is obvious that the code in green leaves the web application vulnerable to injection attacks.
To determine the number of columns, red team members can use the ORDER BY clause.
To remove any unwanted part of the original query, red team members can use double dash.
Finally, finding the correct table names is trivial, since the osquery schema is publicly available.
In addition, when responding to an incident or proactively hunting, blue teams are oftentimes not thorough enough. For example, if any of the “classic” osquery queries that look for active connections returns empty, no further investigations are performed . Red team members can tamper with a system’s kernel post-exploitation and bypass the TCP/IP stack of a system, this way they can establish covert C2 communications that will remain undetected by the aforementioned “classic” osquery queries. They can do so using tools like RedSails or PeddleCheap’s packetredirect module (against Windows systems).
Credits to defensivedepth.com whose posts, cheat sheets and handouts introduced me to osquery and its capabilities in the first place! Go check them out…
