Round 2 of ATT&CK Evaluations is Now Open

Last year we launched ATT&CK Evaluations to bring transparency to evaluations of security products, empower end-users with objective insights, and drive the security community to improve capabilities. We worked collaboratively with vendors and approached these evaluations as peers. With those goals in mind, in November 2018 we publicly released both our test methodology and the results of our initial cohort of seven vendor evaluations for Round 1. We followed the initial cohort with “rolling admissions” evaluations, the first of which were released in February 2019.

Round 2 of ATT&CK Evaluations will begin in Summer 2019 and will be based on the threat group commonly known as APT29/COZY BEAR/The Dukes. While our goals for Round 2 remain the same as Round 1, we are changing some details of our approach to improve and build on the success of Round 1, as we discuss below.

APT29/COZY BEAR’s notoriety from major breaches along with their sophisticated techniques make them an ideal follow-on emulation to contrast with our Round 1 emulation. With APT3/GOTHIC PANDA, we focused more on noisier, process-level techniques, such as using built-in operating system utilities. In comparison, APT29/COZY BEAR is known to use more sophisticated implementations of techniques through custom malware and alternate execution methods, such as PowerShell and WMI, which we plan to emulate in Round 2. Our emulation of APT29/COZY BEAR will intentionally be different than the emulation of APT3/GOTHIC PANDA, and we are excited to see how detection capabilities stack up against these changes in emulated behaviors.

In addition to a new adversary, we are making several other changes to our approach.

First, we are adding community contributions to enhance our emulation of APT29/COZY BEAR. Our recently-closed public call for threat intelligence seeks to enable a more community-driven approach towards developing ATT&CK Adversary Emulation Plans as we create our APT29 methodology. Through community contributions we hope to ensure that our evaluations are accurate representations of the threat, which we hope will better inform the community about whether defenses can address them.

Secondly, in Round 1, we divided the evaluations into an “initial cohort” of vendors who executed contracts by a certain date and subsequent “rolling admissions” for vendors who executed after that date. As we move towards the future of ATT&CK Evaluations, we intend to not offer rolling admissions for Round 2, and we are ending rolling admissions for Round 1. There are a number of factors that led to this decision, chief among them is ensuring we can focus our efforts on a single round to provide the best product for vendors and end-users.

Lastly, we will be updating our detection categories based on the feedback we’ve gotten and what we learned in Round 1. We hope this will reduce confusion and make the results easier to understand. We will release more information about the new categories in the coming weeks. We continue to strive to identify ways to discuss various types of detections across different vendors in a common way, while still showcasing their differences in approaches and capabilities.

We look forward to building on Round 1 and improving our approach for Round 2. For more information on Round 2 participation, please contact attackevals@mitre.org.

©2019 The MITRE Corporation. ALL RIGHTS RESERVED Approved for public release. Distribution unlimited 18–03621–10.