Mapping ATT&CK Data Sources to Security Events via OSSEM 🛡⚔️

Published in

Open Threat Research

11 min readOct 28, 2020

The MITRE-ATT&CK team just released the last entry of a two-part blog series where they proposed a new methodology to start defining and extending the concept of ATT&CK data sources. They went from identifying opportunities and enhancing the current state of data sources, all the way to operationalizing a methodology to improve the definition of current data sources and the identification of new ones within the framework. We highly recommend you to read those blog posts first 😉 .

One of the main changes in the ATT&CK framework, from a data perspective, is the new structure applied to the data source objects concept. Data sources went from being simply additional metadata for each (sub)-technique to its own ATT&CK object with its own attributes as shown below:

From a contribution perspective, we believe that the InfoSec community can start not only by improving current data sources definitions and proposing new ones, but also to start mapping and sharing security telemetry related to relationships among data elements. According to the ATT&CK team:

The diagram below maps out how ATT&CK could provide information from the data source to the relationships identified among the data elements that define the data source. It’d then be up to you to determine how best to map those data components and relationships to the specific data you collect

In this blog post, we the Open Threat Research (OTR) community, will share our approach on mapping a few security events to relationships among data elements through open source projects such as Open Source Security Event Metadata (OSSEM), Mordor, and the Threat Hunter Playbook. We will show an extended yaml file with security event logs mapped to specific data relationships, and how we have applied it when developing detection analytics for simulated adversary behaviors.

What is OTR? 🧐

At the beginning of the year, a community based effort, Open Threat Research (OTR), was founded to empower the InfoSec community through open source projects and bring other passionate threat researchers together to collaborate and share. Overall, we hope everything that is contributed through the OTR community can help organizations to improve their security posture.

One good example of the type of open source contributions developed under the OTR community is the OSSEM project. A project that has helped a lot to the revamping of data sources in the MITRE ATT&CK framework. We are very happy to see those contributions impacting a world-wide community leveraging ATT&CK to validate their detection rules and improve their data collection strategy.

What is OSSEM?

The Open Source Security Events Metadata (OSSEM) is a community-led project that focuses primarily on the documentation and standardization of security event logs from diverse data sources and operating systems. Security events are documented in a data dictionary (DD) format and can be used as a reference while mapping data sources to data analytics used to validate the detection of adversarial techniques. In addition, the project provides a common data model (CDM) that can be used for data engineers during data normalization procedures to allow security analysts to query and analyze data across diverse data sources. Finally, the project also provides documentation about the structure and relationships identified in specific data sources to facilitate the development of data analytics and adversary behavior representation from a data perspective. This relationships are stored under the detection model (DM) section of the project.

Open Source Security Events Metadata (OSSEM) project general schema

How Do We Start Mapping ATT&CK Data Sources to Security Events? 🤔

Within its GitHub repo, the ATT&CK team shared their initial data sources objects in yaml format. Based on this initial file, we can extend the data sources objects and map them to security telemetry such as Windows Security Auditing, Windows System, Sysmon and PowerShell.

The process is pretty straight-forward and involves the following steps:

Understanding security event logs via data dictionaries
Identifying relationships in security events logs
Mapping security event logs to ATT&CK data sources objects

Understanding security event logs via Data Dictionaries

In order to understand and map a specific security event logs to a data source object, we needed to comprehend the structure and data provided by every event log. We believe that the best way to accomplish this is documenting security event logs through data dictionaries.

The Data Dictionaries (DD) sub-module of the OSSEM project gives you information about telemetry that can be colledted from different platforms. If you want to consume OSSEM data dictionaries in a programmatically way, you can find the yaml files here.

Microsoft Windows Security Auditing — Event 4697

Identifying relationships in security events logs

If you are trying to map security events to already defined relationships in the ATT&CK data source objects, here is where you can try to validate if those relationships exist in your security events. For example, the new data source object “Service” has the following relationship:

user created a service

Looking at the previous image, data dictionary for event 4697 from Windows Security, we can validate that the “SubjectUserName” and “ServiceName” data fields give me context that represents this relationship. Therefore, we can say that the Windows Security event 4697 is a good candidate to be mapped against the following relationship “user created a service”. If the relationships identified in the security event logs you are working with does not exist in the framework, then it is a good opportunity to contribute to the ATT&CK data source objects project.

Mapping security event logs to ATT&CK data sources objects

Once we have a better understanding of the data provided by security event logs and the relationships identified in them, we can start the mapping them to data sources objects. In the image below, we are showing an example of the initial mapping performed over the “Service” data source object.

Some of the data fields that we are adding to the original YAML file are the following:

relationship name
relationship id
telemetry
event_provider
event_id

You can find our initial mapping in yaml format within the Detection Model (DM) sub-module of the OSSEM project.

Service data source object — Security event logs mapping

Within the OSSEM-DM sub-module, you can also find a quick reference of all the current security event logs mapped to ATT&CK data sources objects in this markdown file.

Mapping of data sources to security event logs

How Can We Consume the Security Event Logs Mapping from OSSEM? 🧐

We have prepared a Jupyter Notebook for you to explore the security telemetry mapped to new ATT&CK data sources objects. In this notebook, we are also sharing with you a couple of ideas on what you can do with the information provided by the OSSEM project. You can find the notebook here.

Visualizing relationships among data elements using a Network Graph

Without modifying the original ATT&CK data sources yaml file, we can start analyzing its information from a visualization perspective. Here is where the concepts such as source_data_element, relationship, and target_data_element become really important.

According to the first part of the ATT&CK data sources blog series, adversary behavior might be represented using data elements and relationships. Therefore, it makes sense to represent the interaction among data elements using a network visualization. In the image below, we show a basic example using the Bokeh Python library. We can see that the user and process data elements are the ones that concentrate most of the interactions.

Mapping (sub)techniques to potential security event logs

Another interesting use case when analyzing the initial mapping from OSSEM is actually to complete the mapping process all the way to (sub)techniques.

Within the jupyter notebook that we have prepared for you, we are using the attackcti Python library to collect and parse the content of the ATT&CK framework. In addition, we are updating our dataframe with the new data sources names proposed by the ATT&CK team. Finally, we were able to start mapping security event logs to (sub)techniques. The example that we present at the end of the notebook is the mapping process for the SMB/Windows Admin Shares (T1021.002) sub-technique. This mapping example only considers the current recommended data sources by the ATT&CK framework.

Mapping of security event logs/relationships/components to SMB/Windows Admin Shares

How Can We Operationalize the Security Event Logs Mapping? 🤔

The idea of revamping the data sources piece of ATT&CK was to get the framework closer to the community and support the representation of adversary behavior, development of detection analytics and improve our security posture from a data perspective. The security event logs mapping output from the OSSEM project is a very helpful resource that we can use to support all these activities. A practical way to show you its application, is the development of detection analytics for Mordor datasets through the Threat Hunter Playbook.

Let’s continue using the SMB/Windows Admin Shares (T1021.002) sub-technique that adversaries may use to accomplish lateral movement. The Mordor project contains a small dataset that we can use to study this specific behavior, so we were able to put together a playbook where we share our experience developing detection analytics.

The offensive tradecraft behind the sub-technique simulation is that adversaries might be using the Server Message Block (SMB) protocol to copy files over the network to either execute code remotely or exfiltrate data.

Some of the actions that the adversary developed in our environment are accessing the file share, accessing the file, and copying or creating a file. Using data elements and relationships provided by the ATT&CK team and the data sources mapping from the OSSEM project, we can represent this behavior with the following schema:

Potential data representation of adversary behavior for SMB/Windows Admin Shares (T1021.002) sub-technique

Using this adversary behavior data model, we can start defining detection analytics after analyzing the Mordor dataset.We might not use all the relationships from this data model, however all of them are a very good reference when trying to get more context of the adversary behavior. Within the playbook, you will find at least 5 detection analytics related to the SMB/Windows Admin Shares (T1021.002) sub-technique. In the following image, a couple of these basic detection analytics are shown:

Detection analytics II for SMB/Windows Admin Shares (T1021.002) — Threat Hunter Playbook

Detection analytics V for SMB/Windows Admin Shares (T1021.002) — Threat Hunter Playbook

After performing this mapping exercise, we were able to identify a potential contribution opportunity to the MITRE-ATT&CK framework from a data sources perspective. The current data sources recommended are Authentication log and Process. However, when defining our detection analytics, we required more security context around creation and manipulation of a file. Therefore, we are proposing to consider also the File data source object as part of the recommended data sources for this sub-technique.

Contribution opportunity — File data source object as recommended data source

Wait… Is the Threat Hunter Playbook applying the Security Event Logs Mapping from OSSEM? 😱

The answer is YES!! 🍻 😉 Since the beginnings of the project, we considered data sources as a main component of the project. However, we decided not to show it as part of the playbooks because we were looking for a methodology that allows us to start mapping security events in a more effective way. After the release of ATT&CK data sources objects, we have updated all the playbooks within the project with data sources metadata. This metadata will help you to get a better understanding of the logic behind the detection analytic. You can find and review all the available playbooks here.

SMB Create Remote File example — Threat Hunter Playbook

How can you contribute? 💜

a) Fork the Detection Model (DM) sub-module of the OSSEM project

You can use this GitHub document as a reference to understand the forking a GitHub repo process. You will find tabs with content for Mac, Windows and Linux platforms.

b) Access the attack_event_mapping section

Our preferred application to handle the documentation of the project is Visual Studio Code, but you can use the one that works better for you. You will need to access the attack_event_mapping section where we have made available a yaml file per data source object. As it was described before, these yaml files are just extensions of the original ones provided by the ATT&CK team. For example, in the image below you can see the Logon Session data source object.

logon_session data source object example

c) Share With The Community

Adding security events to current relationships (Option #1)

You can start by adding security events to already existing event providers like Microsoft-Windows-Security-Auditing, or you can also add a new event provider to the relationships defined by the MITRE ATT&CK team.

Example — Adding security event to **User created Logon Session** relationship

Adding a new relationship/component (Option #2)

Also, based on the methodology described by the AT&CK team, you could define new relationships and data components when learning about security events. This type of contribution would be really awesome because you are not only contributing to the security events mapping community effort, but also helping the community to document and propose new content for the MITRE-ATT&CK framework. I am one of the ATT&CK members that developed the methodology, so I can provide feedback before it is contributed to the framework itself via the attack-datasources GitHub repo

Example — Adding a **new relationship/data component**

d) Submit a Pull Request (PR) to the Detection Model (DM) sub-module of the OSSEM project

You can use this GitHub document as a reference to understand the creating a pull request process.

That’s it for this post! I hope you found it useful for the understanding and application of the new methodology proposed by the MITRE-ATT&CK team. We will continue sharing practical applications of this methodology when conducting detection research through OSSEM, Mordor, and the Threat Hunter playbook. Please feel free to review the projects here and submit a pull request. There are a lot and diverse telemetry sources that need to be mapped to (sub)techniques, and we can not do this without your help 😉 🍻 🌎 💜.