Automate the Creation of ATT&CK Navigator Group Layer Files with Python 🐍
What do you do when you have a few hours of free time? Well, besides working out or watching a favorite TV series over and over, you create new API functions for one of your favorite Python projects right? 😆. That’s the case for me with the ATT&CK Python Client project. A project where I host a Python library called attackcti which was developed to help others in the community to interact with ATT&CK content through STIX/TAXII 2.0 APIs and some Python code.
In this post, I will show you how you can programmatically create ATT&CK Navigator threat actor layers in the form of JavaScript Object Notation (JSON) files for every single group available in ATT&CK.
Isn’t that already done?
If you had not noticed it yet, ATT&CK Navigator actor layers are already provided for each group in the ATT&CK site (i.e APT12 Group)
if you click download, you get the following file:
and if you click on view, you get to the following Navigator view:
However, while creating a new function for the attackcti library to export every single technique being used by every single group documented by the ATT&CK team, I figured I could take every single relationships between a group and techniques and automate the creation of each navigator layer file without going through every single ATT&CK page and download every single file manually.
What is attackcti?
It is a python library developed as part of the ATTACK-Python-Client project that I started last year (2018) and that I use to access up to date ATT&CK content available in STIX format via a public TAXII server. This project leverages python classes and functions from the cti-python-stix2 and cti-taxii-client libraries developed by MITRE.
I had never heard of the STIX/TAXII 2.0 APIs before so I decided to start the ATTACK-Python-Client project to learn more about it. After writing a few scripts to interact with ATT&CK’s public TAXII server, I decided to write my own python library. For more information about the development of the library, I highly recommend to read the attackcti library documentation.
Install attackcti
You can install the library via pip as shown below
pip install attackcti
New attackcti function?
I wanted to create a function to loop through all the documented groups in ATT&CK and extract the techniques used by each group. The python library already has a function called get_techniques_used_by_group() which I use to retrieve all techniques used by one specific group. However, when I use the function to loop through every single group (around 90), it takes a very long time to finish (Too many API calls? maybe?).
Therefore, I decided to create a new function named get_techniques_used_by_all_groups() using a more efficient approach to retrieve techniques metadata from more than 90 groups available in ATT&CK in a shorter period of time.
What do you mean 😆?
In order for me to get techniques used by a specific group, I need to retrieve STIX objects that contain those relationships. However, making a request to ATT&CK’s TAXII server with specific relationship filters is not as fast as I would like it to be when performed in a loop through more than 90 groups available in ATT&CK.
How does a Group-Technique relationships work?
In STIX format, you need to find relationship STIX objects of relationship type “uses” which define an intrusion-set (Group) as the source and an attack-pattern (Technique) as the target. For example, if I want to find techniques used by the APT12 Group, I will need to find relationship STIX objects like this one:
Relationship object
The relationship object contains a source_ref field that points to an intrusion-set and a target_ref field that points to an attack-pattern object.
There is also a field named description for most of the relationship objects. The one from the APT12 relationship above states the following:
[APT12](https://attack.mitre.org/groups/G0005) has used blogs and WordPress for C2 infrastructureIn STIX format, I would translate it to :
Group APT12 (intrusion-set — c47f937f-1022–4f42–8525-e7a4779a14cb) accomplishes its Command-and-Control (Tactic) objective by leveraging external web services (Technique - attack-pattern — 830c9528-df21–472c-8c14-a036bf17d665) such as blogs and WordPress (Procedures).You can take the relationship object and use the source_ref and target_ref values to retrieve specific intrusion-set and attack-pattern objects.
Intrusion-set object
Attack-pattern object
Intrusion-set? Group? Attack-pattern? Technique?
Before we continue, I believe it is important to understand how ATT&CK objects and properties are represented in STIX 2.0 format. I will be using intrusion-set and attack-pattern a lot through this post.
How do I query relationships in ATT&CK?
Once again, I highly recommend to first read the docs I put together about cti-taxii-client and cti-python-stix2 libraries. Those two summarize several of the concepts that I had to read to understand how to perform a simple query against ATT&CK’s TAXII server.
Import STIX and TAXII Libraries
from stix2 import TAXIICollectionSource, Filter, CompositeDataSource
from taxii2client import CollectionSet ATT&CK TAXII Collection ID Variables
The public ATT&CK TAXII instance has three main collections (Enterprise, PRE and Mobile). Every collection has an ID which attackcti uses to retrieve ATT&CK STIX objects from all those matrices.
ATTCK_STIX_COLLECTIONS = "https://cti-taxii.mitre.org/stix/collections/"ENTERPRISE_ATTCK = "95ecc380-afe9-11e4-9b6c-751b66dd541e"
PRE_ATTCK = "062767bd-02d2-4b72-84ba-56caef0f8658"
MOBILE_ATTCK = "2f669986-b40b-4423-b720-4396ca6a462b"
Initialize TAXII Collection Sources
According to STIX2 docs, the TAXIICollectionSource API provides an interface for searching/retrieving STIX objects from a local/remote TAXII Collection endpoint. In our case, we are pointing to our ATT&CK TAXII Collection instances ( https://cti-taxii.mitre.org/stix/collections/<Collection ID>)
ENTERPRISE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + ENTERPRISE_ATTCK + "/")
TC_ENTERPRISE_SOURCE = TAXIICollectionSource(ENTERPRISE_COLLECTION)PRE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + PRE_ATTCK + "/")
TC_PRE_SOURCE = TAXIICollectionSource(PRE_COLLECTION)MOBILE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + MOBILE_ATTCK + "/")
TC_MOBILE_SOURCE = TAXIICollectionSource(MOBILE_COLLECTION)
Initialize a Composite Data Source
According to STIX2 docs, a user can have a single CompositeDataSource as an interface to a set of DataSources. When an API call is made to the CompositeDataSource, it is delegated to each of the (real) DataSources that are attached to it. In our case, we have three TAXIICollection sources (Enterprise, PRE and Mobile) as defined in our previous step. Therefore, we can use the CompositeDataSource class and the add_data_sources method to attach every ATT&CK TAXIICollection source and be able to query all of them at the same time.
COMPOSITE_DS = CompositeDataSource()
COMPOSITE_DS.add_data_sources([TC_ENTERPRISE_SOURCE, TC_PRE_SOURCE, TC_MOBILE_SOURCE])Retrieve All Relationships
Now that we can query all the ATT&CK TAXIICollection sources at once, we can use the query method and a set of filters to retrieve STIX objects of type relationship. Remember the APT12 relationships file?
Therefore, we can run the following query to retrieve all relationships from every single ATT&CK TAXIICollection source:
rels = COMPOSITE_DS.query(Filter("type", "=", "relationship"))I already automate all the code from above with the already available function named get_relationships() .
from attackcti import attack_client
lift = attack_client()all_relationships = lift.get_relationships()
What about all relationships from a specific STIX object?
What if you want to be very specific and retrieve relationships from a specific STIX object? You can use the relationships method from the CompositeDataSource class to retrieve relationships involving a given STIX object. That is totally different from the query we used before where we got all relationships available in ATT&CK:
rels = COMPOSITE_DS.query(Filter("type", "=", "relationship"))The query below will look for any relationship of relationship type “uses” involving a specific STIX object only.
rels = COMPOSITE_DS.relationships(stix_object, 'uses', source_only=True)I also automate this process with an already available function named get_relationships_by_object() . You still have to provide an STIX object for it to work. You can use an intrusion-set object and retrieve all its relationships.
from attackcti import attack_client
lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)group_relationships = lift.get_relationships_by_object(groups[0])
You will get every relationship of relationship type “uses” involving the specific intrusion-set. The function returns relationship STIX objects similar to the one below:
The source_def points to an intrusion-set, but the target_ref values could be a tool, malware or attack-pattern objects.
How do I retrieve techniques used by a group by leveraging the relationships retrieved?
In that case, we want relationship objects that have target_ref values of type attack-pattern. Following the manual code I shared above, and the results from the get_relationships_by_object() function, you can simply query the ATT&CK Enterprise TAXIICollection source with the filter below:
filter_objects = [
Filter('type', '=', 'attack-pattern'),
Filter('id', '=', [r.target_ref for r in group_relationships])
]techniques_used = TC_ENTERPRISE_SOURCE.query(filter_objects)
As I mentioned at the beginning of this post, I already automate all the steps explained so far via the already available function named get_techniques_used_by_group() .
That function follows the following process:
- Retrieves relationship STIX objects of relationship type “uses” from a specific intrusion-set.
- Retrieves attack-pattern objects looping through all the relationships retrieved and uses the target_ref values in each relationship to pass a filter to ATT&CK’s TAXII public server.
You can simply run the following lines of code and you will be able to retrieve all the attack-patterns used by one specific intrusion-set:
from attackcti import attack_client
lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)group_techniques = lift.get_techniques_used_by_group(groups[0])
What happens when I try to retrieve all techniques used by ALL groups?
You can apply the same get_techniques_used_by_group() function, but against all the groups STIX objects that the get_groups() function retrieves. You can do a simple for loop over more than 90 groups as shown below:
from attackcti import attack_client
lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)def get_techniques_used_by_all_groups():
techniques_used = []
for group in groups:
techniques_used.append(lift.get_techniques_used_by_group(group))
return techniques_used%time techniques_used_by_groups = get_techniques_used_by_all_groups()
However, it takes longer than what I would like it to take. I calculated the time by packing the code in a function and used the Jupyter Notebook %time built-in magic command as shown below:
I decided to measure how much time each loop was taking so I added the Jupyter Notebook %time built-in magic command to each get_techniques_used_by_group() execution.
def get_techniques_used_by_all_groups():
techniques_used = []
for group in groups:
%time techniques_used.append(lift.get_techniques_used_by_group(group))
return techniques_used
With an average of 3-4 seconds per get_techniques_used_by_group() execution times more than 90 groups, it makes sense that the overall time is around 4–5 minutes for the overall function. Once again, it is fine when it is done to one group, but not as fast when it is applied to more than 90 groups.
What if we collect all the relationships at once?
Something that I am learning as I write scripts to interact with ATT&CK’s TAXII server is that the less API requests you provide to get the same results, the better. For example, retrieving relationships for each individual intrusion-set and getting information about each specific attack-pattern involved with the intrusion-set takes around 5 minutes. This is because I am running the following two things around 90 times and waiting for the server to handle each request individually one after the other one:
all_relationships = lift.get_relationships_by_object()
filter_objects = [
Filter('type', '=', 'attack-pattern'),
Filter('id', '=', [r.target_ref for r in all_relationships])
]
TC_ENTERPRISE_SOURCE.query(filter_objects)Therefore, I decided to first collect all relationship objects of relationship type “uses” from every STIX object available in TAXII all in one API request. I measured how much it would take to do that, and I was impressed by how much less time it took. As you can see below, it took 5.56 seconds to get all the relationships at once, versus 4-5 seconds for each individual API request to get relationships from only one STIX object. 😱
from stix2 import TAXIICollectionSource, Filter, CompositeDataSource
from taxii2client import CollectionATTCK_STIX_COLLECTIONS = "https://cti-taxii.mitre.org/stix/collections/"
ENTERPRISE_ATTCK = "95ecc380-afe9-11e4-9b6c-751b66dd541e"
PRE_ATTCK = "062767bd-02d2-4b72-84ba-56caef0f8658"
MOBILE_ATTCK = "2f669986-b40b-4423-b720-4396ca6a462b"ENTERPRISE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + ENTERPRISE_ATTCK + "/")
TC_ENTERPRISE_SOURCE = TAXIICollectionSource(ENTERPRISE_COLLECTION)PRE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + PRE_ATTCK + "/")
TC_PRE_SOURCE = TAXIICollectionSource(PRE_COLLECTION)MOBILE_COLLECTION = Collection(ATTCK_STIX_COLLECTIONS + MOBILE_ATTCK + "/")
TC_MOBILE_SOURCE = TAXIICollectionSource(MOBILE_COLLECTION)COMPOSITE_DS = CompositeDataSource()
COMPOSITE_DS.add_data_sources([TC_ENTERPRISE_SOURCE, TC_PRE_SOURCE, TC_MOBILE_SOURCE])filters = [
Filter("type", "=", "relationship"),
Filter('relationship_type','=','uses')
]
%time all_relationships = COMPOSITE_DS.query(filters)
How do I find “intrusion-set uses attack-pattern” relationships in our results?
Once I collect all relationships, I filter them to only keep relationships that have intrusion-set objects as the source and attack-pattern objects as the target in every relationship object. Remember?
In order to determine if source_ref and target_ref values are of specific types, I use a function from the stix2.utils library named get_type_from_id.
from stix2.utils import get_type_from_idgroup_relationships = []
for rel in all_relationships:
if get_type_from_id(rel.source_ref) == 'intrusion-set'\
and get_type_from_id(rel.target_ref) == 'attack-pattern':
group_relationships.append(rel)len(group_relationships)
print(group_relationships[0])
one relationships result example:
Do we retrieve all groups and techniques at once too? How do I get all techniques used by all groups?
Now that we have all the relationships defining what attack-pattern objects are used by every single intrusion-set, I need to get more metadata about each technique and group. All we have so far in each relationship STIX object is the source_ref field pointing to an intrusion-set ID and target_ref field pointing to an attack-pattern ID. Instead of retrieving more metadata about attack-pattern STIX objects defined on each relationships that involves intrusion-set objects, I decided to retrieve all the available intrusion-set and attack-pattern STIX objects and do a basic match locally.
Get all groups and techniques
Using available attackcti functions, I can complement my manual code and get all the groups and techniques available in ATT&CK as STIX objects.
from attackcti import attack_client
lift = attack_client()groups = lift.get_groups()
techniques = lift.get_techniques()
Match Group -> Relationships source_ref ID
Next, I take all the group_relationships results I got before, and look for the specific intrusion-set ID in the groups STIX objects list. Once there is a match, I create a new python dictionary joining information about the relationship and the intrusion-set objects. The most important additional metadata is the target_ref field which points to a specific attack-pattern ID involving the intrusion-set. The results are then added to a new list named group_techniques_ref as shown below:
import jsongroup_techniques_ref = []for g in groups:
for rel in group_relationships:
if g['id'] == rel['source_ref']:
gs = json.loads(g.serialize())
gs['technique_ref'] = rel['target_ref']
gs['relationship_description'] = rel['description']
gs['relationship_id'] = rel['id']
group_techniques_ref.append(gs)
Match Attack-patterns -> New Dictionary target_ref ID
I apply the same concept as before, and just loop through all the attack-pattern objects and look for the specific attack-pattern id in the new dictionary we created with information about intrusion-set and relationships objects. Once there is a match, I add additional information from the attack-pattern itself to the python dictionaries in the group_techniques_ref list. The results then get added to a new list named groups_use_techniques.
groups_use_techniques = []
for gt in group_techniques_ref:
for t in techniques:
if gt['technique_ref'] == t['id']:
tactic_list = list()
for phase in t['kill_chain_phases']:
tactic_list.append(phase['phase_name'])
gt['technique'] = t['name']
gt['technique_description'] = t['description']
gt['tactic'] = tactic_list
gt['technique_id'] = t['external_references'][0]['external_id']
gt['matrix'] = t['external_references'][0]['source_name']
if 'x_mitre_platforms' in t.keys():
gt['platform'] = t['x_mitre_platforms']
if 'x_mitre_data_sources' in t.keys():
gt['data_sources'] = t['x_mitre_data_sources']
if 'x_mitre_permissions_required' in t.keys():
gt['permissions_required'] = t['x_mitre_permissions_required']
if 'x_mitre_effective_permissions' in t.keys():
gt['effective_permissions'] = t['x_mitre_effective_permissions']
groups_use_techniques.append(gt)
groups_use_techniques[0]The result is a list of python dictionaries with information about an intrusion-set and attack-pattern based on a relationship of relationship type “uses”. In other words, a group uses this technique 🍻. You can see one example of the custom dictionary result below:
Finally, I took everything we have done so far and created a new function named get_techniques_used_by_all_groups(). All you have to do now is:
from attackcti import attack_client
lift = attack_client()
%time techniques_used = lift.get_techniques_used_by_all_groups()How long does it take to run all that now?
let’s test it! 😱 around12 seconds! from 5 mins to 12 seconds! 🍻
What does it all have to do with ATT&CK Navigator?
Remember the Navigator group layer file that you could download from the ATT&CK site? Well, with the new function named get_techniques_used_by_all_groups() you can easily retrieve all intrusion-set object and all the attack-pattern objects related to it in a few seconds. Therefore, it should be easy to build similar navigator layer files with the results of the new function! 🍻
We need a template!
You can take the Navigator group layer file from above and use it as a template to loop through every singe intrusion-set:
Automating creation of Navigator group layers
All I need is the get_techniques_used_by_all_groups() function, some additional lines of code and the template.
Get techniques used by all groups
from attackcti import attack_clientlift = attack_client()techniques_used = lift.get_techniques_used_by_all_groups()
Create a list of group dictionaries
To make things easier, I create a list of dictionaries where each group name is the main key and the value is an empty list where I would append every single technique involving that group.
groups = lift.get_groups()
groups = lift.remove_revoked(groups)groups_list = []
for g in groups:
group_dict = dict()
group_dict[g['name']] = []
groups_list.append(group_dict)groups_list[89]
Group techniques by groups
We can then use the output of the get_techniques_used_by_all_groups() function and start appending techniques to the dictionaries with the key name that matches the group name. If there is a match, I create a new dictionary with specific information that I could use for my Navigator files.
for group in groups_list:
for group_name,techniques_list in group.items():
for gut in techniques_used:
if group_name == gut['name']:
technique_dict = dict()
technique_dict['techniqueId'] = gut['technique_id']
technique_dict['techniqueName'] = gut['technique']
technique_dict['comment'] = gut['relationship_description']
technique_dict['tactic'] = gut['tactic']
technique_dict['group_id'] = gut['external_references'][0]['external_id']
techniques_list.append(technique_dict)
groups_list[89]
Make a dynamic template
We can take the template I shared before, and make it dynamic one to loop through the dictionaries in the groups_list list . This allows me to create a navigator group layer file for each intrusion-set.
Dynamic Template:
Run Dynamic Template
We can then use the template in a loop as shown below:
import jsonfor group in groups_list:
for k,v in group.items():
if v:
actor_layer = {
"description": ("Enterprise techniques used by {0}, ATT&CK group {1} v1.0".format(k,v[0]['group_id'])),
"name": ("{0} ({1})".format(k,v[0]['group_id'])),
"domain": "mitre-enterprise",
"version": "2.2",
"techniques": [
{
"score": 1,
"techniqueID" : technique['techniqueId'],
"techniqueName" : technique['techniqueName'],
"comment": technique['comment']
} for technique in v
],
"gradient": {
"colors": [
"#ffffff",
"#ff6666"
],
"minValue": 0,
"maxValue": 1
},
"legendItems": [
{
"label": ("used by {}".format(k)),
"color": "#ff6666"
}
]
}
with open(('{0}_{1}.json'.format(k,v[0]['group_id'])), 'w') as f:
f.write(json.dumps(actor_layer))
Output Templates
That’s it! As you can see on the left of my Jupyter lab server, I was able to automatically create a Navigator layer file for every group that had techniques mapped to them (Group uses techniques).
Visualizing Navigator Group Layer Files
All you have to do now is download one of the navigator group layer files
Go to the ATT&CK Navigator site: https://mitre-attack.github.io/attack-navigator/enterprise/
Click on the (+) icon to the right of the default “layer” tab and select Open Existing Layer > upload from local
Once you select group layer file, you should get the following view:
If I check the ATT&CK site, we can see the 4 techniques being used by the specific group and I can also check the ATT&CK Navigator view option to validate I get the same results.
Can we create a new function and export all groups navigator layer files? 😆
Yes, I took all the final steps and created another new function named export_groups_navigator_layers() and made it available in the attackcti library. Therefore, all you have to do now to replicate all the code in this post is run the following 3 lines. You will get every single group Navigator layer file in around 10–12 seconds! 😱 🍻
from attackcti import attack_client()lift = attacK_client()%time lift.export_groups_navigator_layers()
That’s it! 🍻 Now, It is up to you how you can take those concepts and the new available functions to improve the creation of Navigator group layers. For example, you can add more properties such as data sources to each attack-pattern cell in the Navigator view.
If you are using a version of attackcti that does not have these latest updates, just run the following:
pip install --upgrade attackctiAlso, I put together a Jupyter Notebook for you to use and go through all the code presented in this post 😉 🍻You do not have to build your own Jupyter Notebook server to run it, you can access it via the Binder link below 😉:
- Notebook name: export_groups_navigator_layers
- Noteebook nbviewer: https://nbviewer.jupyter.org/github/hunters-forge/ATTACK-Python-Client/blob/master/notebooks/export_groups_navigator_layers.ipynb
- Notebook Binder: https://mybinder.org/v2/gh/hunters-forge/ATTACK-Python-Client/master
I hope you enjoyed this post! Working on the attackcti library has helped me a lot to learn more about STIX/TAXII 2.0 APIs for ATT&CK. Therefore, I highly recommend to start using it and explore the code behind it. I am sure there is a lot that can be improved in attackcti, so if you would like to continue the conversations and help contribute to it with some other cool use cases, I recommend to join a slack channel that I host for the community:
Automatic free slack invitation: https://launchpass.com/threathunting
References
https://pypi.org/project/attackcti/
https://attackcti.readthedocs.io/en/latest/
https://mitre-attack.github.io/attack-navigator/enterprise/
https://github.com/oasis-open/cti-python-stix2
https://github.com/oasis-open/cti-taxii-client
https://github.com/mitre/cti/blob/master/USAGE.md#mapping-concepts
