Community Evaluating Free Telemetry 💸 🌎 Following the ATT&CK Evals Methodology ⚔️

Current Telemetry Detection Category Coverage — APT29 Scenario One Only

ATT&CK Evaluations? Tell me more.. (Simplified)

  • Identify the adversary that will be emulated
  • Build out the adversary profile through open source threat intelligence
  • Map adversary behavior to ATT&CK techniques
  • Develop the emulation plan (Step-by-step playbook broken down to the procedure level)
  • Develop a baseline and detection criteria (MITRE team)
  • Execute emulation plan (MITRE team)
  • Capture detections from endpoint vendor
  • Review and validate detections (MITRE team assigns detection categories)
  • Normalize results for consistency across vendors evaluations
  • Produce and release all results

What About a Free Telemetry Evaluation 🤔?

Enter Detection Hackathon

Enter Detection Categories

Main Detection Category

..and Free Telemetry 💸 ?

Day 1 — Host Event Providers

What about the other detection categories?

Conducting a Free Telemetry Evaluation 🤠 ☑️

Prepare Emulation Environment

APT29 — Mordor Labs Environment

Execute Emulation Plan

APT29 Emulation Plan — Scenario 1 (Day 1)
APT29 Emulation Plan — Scenario 2 (Day 2)

Collect Free Telemetry

Host Telemetry Data Pipe —> JSON File

Create Detections (Community Effort ❤️)

Document Detections
Steps > Sub-Steps > Detections

Review and Validate Detections

  1. Validate detections alignment with the detection criteria.
  2. Make sure queries would work 😉: I needed to somehow be able to run those queries against the APT29 datasets in a programmatic way. I will get to it in a little bit ⏰ after showing you how I am building the initial report.

Produce “All Results” Report

ATT&CK Evaluations — Microsoft All Results Table
  • Create detection files for each detection query available in each step file.
  • Create an “All Results” table similar to the image above where I aggregate every single detection with some metadata for a similar look/view.
Step 1.A.1 — Microsoft Detection Notes

Display “All Results” Report

Calculate Coverage Results 💰 (Day 1)

Enter Jupyter Notebooks

  • Loop over every single YAML file that contains detection queries.
  • Use python and nbformat to create a notebook and add every single detection query to it.
  • Download the APT29 dataset programmatically to the notebook and process it as a dataframe to perform additional analysis.
  • Use SparkSQL to query the dataframe (Spark dataframe APIs via python). For the initial round of the detection hackathon, I encouraged participants to share their queries in a SQL-like format (SparkSQL) to expedited this process 😉. We are going to provide a workshop next time (Day 2) to expedite this process and get everyone familiarized with it. That will be a main requirement for Day 2.

How Can I Use All These Resources ? 🚀

All Results Report Table

Run APT29 Evals Notebook
Launching Threat Hunter Playbook’s Jupyter Server
BinderHub Hosted Notebook

That’s It? There is more.. 😆 Contributions!

Rules to be contributed to SIGMA soon.

What’s Next? :

Detection Hackathon — Day 2 ⏳ 🏹

  • Extend the time for the live session from 4 hours to maybe 8 hours but split in two days (weekends). Hackathon can run for days, but the live sessions are key to ask questions and learn more about the process.
  • Provide an initial training to go over Jupyter Notebooks and the basics of Python and PySpark (A free 1–2 hours workshop)
  • Have other SIEMs sponsored or provided by the community to explore the data in a more familiar way meanwhile participants get used to Jupyter Notebooks and PySpark (It is always good to have options)
  • Automate the way how we go from GitHub issues to YAML files (I will try)
  • Live stream music through a separate channel (not the live session call) to allow participants to mute the music if they want to without muting the main communication channel 😉 (Music is a must!)

Educational Material

  • I believe we could also use these resources for training purposes for other query languages besides SparkSQL (i.e Kusto Query Language)
  • Having data mapped to adversary procedures created from open source threat intelligence is a great material to link queries and techniques to one adversary profile. Not just one query mapped to one adversary technique but several queries mapped to a complete behavior or adversary profile.
  • If you get to use this material in your own training, make sure you attribute the work to the Open Threat Research Community!. These are the types of collaboration efforts and projects that this new community movement will prepare and run for the InfoSec community. Stay tuned for more news about it!



Threat Hunting, Data Science & Open Source Projects

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store