Fully Automated Threat Hunting. Too Good To Be True?
Can threat hunting be fully automated?
Spoiler: we don’t know (for now)🙈
We created a poll on LinkedIn and asked what security professionals think about the possibility of full threat hunting automation. Here are the results:
Now it is a perfect opportunity to use the all-time-favorite It’s complicated.
Most of you believe that it’s impossible to automate threat hunting fully. By the way, we appreciate comments under the poll. So insightful! We are thrilled to hear all the voices in cybersecurity. You might have guessed that we love open discussion since we conduct a crowdsourcing initiative and run a Detection as Code platform.
Ok, so the perspective of automated threat hunting is fuzzy. That’s a hypothesis for now. But why? We decided to analyze current threat hunting automation possibilities and their future potential. That’s what we’ve got.
Threat Hunting is Cool
Usually, we want to automate anything boring, redundant, too excessive, and too expensive to do manually. So why would we want to automate threat hunting if it’s the coolest thing that a SOC analyst can possibly do? Truly trailblazing threat hunting occurs in organizations with lots of resources and exceptional proficiency of cybersecurity superstars (a bit lame, we know). It’s a Sherlock Holmes level of excellence, if you know what we mean. If not, let’s look at the subject more closely.
For starters, threat hunting requires deep knowledge of the network. That’s why it’s done mainly by the coolest guys in the SOC hierarchy — tier 2 and tier 3 analysts. The hunter’s job is to assume that some attack has already happened or is happening and create a hypothesis that they either prove or disprove.
What info sources do they use?
- Analytics (behavioral, statistical, technical, etc.)
- TTPs
- Vulnerability feeds
- Threat intelligence feeds
- Vulnerability scanners
- Software composition analysis (SCA) tools
- Adversary emulation environments
- SIEMs
- And more
Now, let the hunt begin!
Every threat hunting process is unique. In a way, it’s like science or even an art form. It starts with intuition, while the flow depends on the brilliance of a hunter. They need to perfectly know the weaknesses of their organization, possessing quite a specific, sometimes proprietary info. Additionally, they should be spies in a way, knowing what happens in the depths of the dark market and other networks around the world (we mean threat intel, nothing illegal🔎).
Can a machine become the next Sherlock Holmes? We’d like to answer it with a question. Do you remember what happened to the Tay bot (in less than 24 hours)?
In general, automation works well if you know what you are looking for. But here, the situation is different. Alerting based on pre-defined conditions contradicts threat hunting because hunting looks for something new and undiscovered. It’s about getting to a zero-day vulnerability faster than the attackers. Imagine finding something not covered in MITRE (which is quite extensive) — it sounds as challenging as proving a string theory. Nevertheless, threat hunters do that, and to date, only human intelligence is capable of performing tasks of such difficulty.
Of course, it isn’t a job for anyone. Finding a good threat hunter is like finding the next Brad Pitt (a bit of cross-industry examples). Anyway, delegating some tasks is a reasonable approach. Whether you trust it to robots or humans — it’s your call.
Threat Hunting Automation Now — In Da Mix
Today’s threat hunting tends to have a mix of manual and automated tools. Automation helps finish time-consuming tasks faster, helping hunters do their job more efficiently.
Examples of the tasks that can be automated:
- Gathering log files and other evidentiary material
- Producing pre-formatted reports based on identified activity
- Automated risk scoring based on threat intelligence, TTPs, and traditional logs
- Advanced search capabilities
- Attack emulation
On the other hand, the following tasks are done best when performed manually:
- Refining and enriching data using analytics
- Using tools and methodologies to investigate
- Revealing new patterns, tactics, techniques, and procedures
Combine both, and the outcome is a unique, unseen threat identified and mitigated ahead of the attackers. Wow!
Automation Examples
A good threat hunter needs to think like an attacker. If we thoroughly study malicious behavior and simulate the environment with available tools, we can prove or disprove our threat hunting hypothesis. For example, certain events typically occur before the attack takes place. You can find common pre-attack patterns and their detailed description by MITRE at this link.
By using MITRE ATT&CK Navigator, you can make queries and download the results in JSON machine-readable format. Additional details on adversary behavior are provided in comments to the JSON code. Then, you can feed this data to your auto-tools.
Also, you can use out-of-the-box solutions like Caldera. It is an open-source tool for emulating adversary behavior that is also widely used in threat hunting. You can download it from GitHub and obtain the needed tools, techniques, visualizations, and simulations (including incident response) for agents running on different operating systems that communicate with Caldera’s core system.
Another example is Atomic Red Team, an ecosystem that performs automated adversary emulation. All the malicious techniques are mapped to MITRE ATT&CK and come in a machine-readable format. As a result, you can integrate this emulation with other tools or create new ones.
Some parts of the research that involve SIEM logs investigation can also be partly automated. It is a point between a threat hunter deciding what data they want to get (how, why) and analyzing the results. Let us give you a “for instance.”
As we all know, it’s impossible to review billions of events by hand. That’s why we need rules. Usually, detection engineers use condition action rules, and many hunters take it from there. However, if you dig deeper, you can also write rules based on arbitrary conditions or anomalous behavior. For example, suppose you apply a negative condition rule to recognize the absence of an event over a specific time. In that case, you can run into quite interesting conclusions, like, for example, identifying a backup process that misses the scheduled routine.
Full Threat Hunting Automation — Yes or No
AI and ML are the current hope for next-level automation. However, they do better when it comes to statistics rather than running into unique conclusions. Also, as our poll participants fairly mention, another big threat on the horizon is quantum computing. But, if attackers can potentially crack encryption, maybe cybersecurity defenders can also use those qubits to do something good instead? Either way, technology innovation is something worth thinking about and experimenting with.
However, right now, due to its limited functionality, automation in no way is comparable to the human brain’s capabilities. If your hypothesis is as simple as Windows on this device was compromised, and the goal of the analysis is to look for any signs of intrusion, automation could work. Then again, hunting implies finding something that your automated machine has no previous knowledge of. That’s why full automation in threat hunting is questionable.
Hunting implies finding something that your automated machine has no previous knowledge of.
So because of this initial confusion between what goes first, the condition, or the outcome, many people confuse threat hunting with other cyber activities. For example, studying the behavior of detected attacks is more about retrospective analysis than hunting. It is something that you actually have, something that you can poke and see what happens, so it’s not hunting. You hunt for something you don’t see and aren’t even sure if it exists.
On the other hand, proactive threat hunting may have too many burdensome manual tasks to perform. If some of them are not automated, finding a new threat is like looking for a needle in a haystack.
