Announcing 2020’s ATT&CK Evaluation: Carbanak and FIN7

Two years ago, MITRE launched ATT&CK Evaluations to drive the enhancement of endpoint detection solutions. This year MITRE’s Public Foundation, MITRE-Engenuity, will take the reins and continue to propel ATT&CK Evaluations forward. The Evaluations empower users to make more informed decisions on endpoint detection capabilities through a transparent evaluation process. Using MITRE’s ATT&CK framework as the benchmark, we articulate how commercial security products can detect adversary behavior. Adversary emulation, testing “in the style of” a specific adversary, allows us to scope the evaluation and ensure our evaluations are informed by known threats.

The first round of ATT&CK Evaluations, emulating APT3, embarked with seven vendors in late 2018. We followed that up with five rolling admission entries, releasing the last of the APT3-themed evaluations in October 2019. The second round, emulating APT29, saw 21 vendors signing up, including 10 new participants. We are preparing the second round results for public release during the March-April timeframe. While we complete the APT29 Evaluations, we’ve started preparing for the next round of evaluations.

For round three, we have chosen to emulate the threat groups that are identified as Carbanak and FIN7. These groups are often conflated, but available threat intelligence indicates that they appear to be multiple threat actor groups that each leverage the Carbanak malware. During our emulation, we will strive to capture what makes the specific groups unique.

Both Carbanak and FIN7 have a well-documented history of widespread impact. Carbanak is cited with the theft of a cumulative $300M (though some estimates are much higher) from hundreds of banks across 30 countries. FIN7 is credited with the theft of more than 15 million customer credit card records from victims spanning over a hundred US companies (across 47 states) as well as international markets. These pervasive threats and their financial motivation will allow us to test novel implementations of familiar techniques as well as techniques unexplored in previous rounds of evaluations.

These groups are infamous for using innovative tradecraft, with efficient surveillance and stealth at the forefront of their strategy. They often rely heavily on scripting, obfuscation, “hiding in plain sight,” and fully exploiting the users behind the machine while pillaging an environment. They also leverage a unique spectrum of operational utilities, spanning both sophisticated malware as well as legitimate administration tools capable of interacting with various platforms (Windows and Linux — including point-of-sale specific technologies). This round will mark the first time Linux endpoint sensors have been included as in-scope for the ATT&CK Evaluations, although the majority of the activity will continue to focus on Windows platforms.

Community Contributions

As always, we welcome your contributions on Carbanak and FIN7 to help inform our emulation plan. With this round, however, we would like to make the process particularly effective by focusing the request. We ask that contributions be limited to the following:

• Do you have clear information about how these two, often conflated groups are differentiated?
• Are there behaviors regarding ATT&CK techniques used by either Carbanak or FIN7 that are not currently captured in the Round 3 Technique Scope?
• Is there any other relevant information that should be considered for an emulation of Carbanak and FIN7 that you believe is novel (i.e. not well documented publicly) yet supported by credible and shareable evidence?

Contributions will be due April 1, 2020 to ensure we have time to review your inputs and integrate them into our emulation plans. The process for sharing will remain unchanged:

• Email us at evals@mitre-engenuity.org with your contribution (If you’d prefer secure means, email us at the above address, and we’ll get back to you with a secure sharing method). Your real name must be included for your information to be considered. Contributions from company accounts may add to the credibility of the information, but we are always happy to accept contributions from independent researchers.
• We are looking for information about the group behaviors as well as the overall way they perform intrusions. Information structured using ATT&CK tactics and/or techniques is helpful, but not required.
• Tell us how you would like to be credited. You can choose to be credited with your name and/or company name, or alternatively, you can choose to remain anonymous. For any anonymous contributors, we will work with you to produce a short statement about the general visibility you have that led to you having access to the information.
• We will not accept any leaked, proprietary, or sensitive information that was not released with the permission of the original source. Contributions are strictly on a voluntary basis for researchers and analysts who wish to share their own information.

Protection Evaluations

The most requested extension to ATT&CK Evaluations is coming: protection evaluations. We’ll only be offering it as an optional extension to our ATT&CK Evaluations detection evaluations. There isn’t a protection-only evaluation, although vendors can still elect to participate in detection-only evaluations, similar to those performed in our other rounds to date.

Endpoint protection technology focuses on stopping adversary behavior before they can achieve their objectives, whereas detection focuses on offering visibility to better assess whether activity is good or bad, as well as providing details on the behavior. Protection and detection offer distinct but complementary technology, and this is an influencing factor in how we are evaluating protection capabilities.

Protection evaluations will focus specifically around protections that address ATT&CK-based behaviors, and we will execute different stages of our scenarios to capture the breadth of protections. Lastly, protections must be built-in automated protections; human-in-the-loop protections will be out of scope. The evaluation process for protections will mirror that of detections. We will execute our emulation, and if we are prevented from progressing, the vendor will demonstrate how the activity was blocked. We will then capture that information, categorizing it similarly to how we assess detections. The protection coverage will be included in the overall evaluation results.

MSSPs Are No Longer Included

Detection categories allow us to abstract our observations from the evaluations and talk about unique solutions and approaches in a common way. As new vendors participate, capabilities evolve, and as we learn how the results are being leveraged, we need to evolve how we talk about them. We made significant changes to detection categories between the APT3 evaluation and APT29 evaluation to address a number of points of confusion. These changes were well received by people looking into the results, and we don’t foresee needing to significantly overhaul the categories again for Round 3, but there will be some slight modifications. We will wait until Round 2 is finished to finalize any changes to the categories, but one thing that we can definitively state is that the MSSP category will no longer be included in our evaluations. While we recognize that MSSPs can bring value to these solutions, our open book and collaborative evaluation design does not lend itself to effectively assessing these capabilities. We will make all protection and detection categories for Carbanak+FIN7 evaluations available near the end of April.

The first two rounds of evaluations have offered us a chance to collaborate with great companies and amazing people from across the industry. We have been able to see how products are evolving to address the threats to our organizations, as well as how their understanding and usage of ATT&CK has evolved. We look forward to continuing this collaboration in the coming year. For more information on participating in the Carbanak+FIN7 ATT&CK Evaluations, please contact evals@mitre-engenuity.org or visit https://attackevals.mitre.org.

© 2020 MITRE Engenuity. Approved for public release.