Chronicle Road to Detection: Approach — Part 2 of 3

Note: this blog series now exists in a cleaned-up full whitepaper form at https://hubs.ly/H0pvD_30

Chronicle has two secret weapons to make our detection approach superior to that of others:

1. We have the structured data (organized via Unified Data Model or UDM) — this means that both rules and algorithms will run reliably and detect cleanly using any data collected by the platform. This includes logs, EDR records, network traffic metadata (from Zeek and friends) and other telemetry — way more than a traditional SIEM tool.

2. We have automatically enriched data (connected to DHCP, DNS, threat intelligence and other context sources) — this means that the detection logic will apply intelligently and not via a blunt text match (e.g. we can match a domain threat indicator to a log that has only an IP address in it! Can your SIEM do that? :-))

From many available approaches, it would be tempting to predict that our detection would rely on machine learning or perhaps even AI. Well, analytics and machine learning may help with revealing anomalies and detecting some of the unknowns.

However, in most cases, defenders do know something about the threat nature, or its behavior. What we need is an easy and effective way to “channel” this knowledge into detections. This means a new approach that utilizes rules and other detection methods is needed.

A good detection engine allows an enterprise to build specific rules for atomic observables, but should also allow for the creation of more general rules to support the study and identification of potential TTPs. By allowing for this continuum of logic, security teams can create detections for key observables such as IOCs and for attacker tradecraft such as specific activity sequence or a process execution patterns.

Enabling both scenarios is critical to the success of both a detection rule engine and the defenders’ security posture.

Can Your Rules Do That?

But how can one be “novel” and “advanced” with rules? Chronicle is coming out with novel detection logic (rules, later models, etc) that has…

• Broad coverage of security telemetry such as logs, endpoint / EDR data, network traffic, metadata, etc
• Petabyte scale for both real-time and historical detections, by running in the cloud on Google infrastructure
• Unified data model (UDM) under the hood enabling consistent data quality and accessibility
• Smart matching logic that enables the system to match to enriched data “automagically”
• Focused detection language that is designed for threat detection, not merely for querying the data
• Allows us to mix and match rules and machine learning models at a later time.

Our approach will also deliver additional benefits such as being easy to understand and test (a challenge that plagues algorithmic and especially ML-based threat detection approaches). The world has enough inscrutable 50-line SQL queries as it is ...

In fact, to enable collaborative detection across the industry, the detection logic needs to be easy to read, review, package and share. Now, that does not mean that anybody can detect an advanced threat using our approach without any training — some hard security problems are, well, hard. It does mean that reading a rule will allow anybody to understand how the detection will work — hence this approach is easier to teach and explain to build security talent at an organization.

Naturally, our method will support both simple detections (such as string matching rules for file names or registry keys) and complex logic (such that may look like an actual script or a program).

For those who will need even more, the approach is extensible and modular. New components can be written and connected to rules and detections to enable new operations in the future. For example, if you reverse the DGA algorithm, you can then drop the logic into a detection rule.

Naturally, the approach will be open and public, as well as mapped to existing frameworks such as MITRE ATT&CK and Sigma.

In the next post, I will introduce the details on YARA-L, our approach to detection. Look for Part 3 tomorrow.

(written together with Brandon Levene)