Building Confidence in Those That Secure Us: Introducing ATT&CK Evaluations for Managed Services

Today we open our first Call for Participation for MITRE Engenuity ATT&CK® Evaluations Managed Services, designed to provide transparent and impartial insights into how managed security service providers (MSSPs) and managed detection and response (MDR) capabilities provide context to adversary behavior. The Call for Participation is open now through February 25th, and prospective participants can visit our site, or email the team for additional information.

Managed services play a pivotal role in today’s security posture. According to preliminary results in our upcoming Managed Services survey, 58% of organizations rely on managed services to either complement their in-house security operations center (SOC), or as their main line of defense. This number jumps to 68% when considering companies under 5,000 employees. At the same time, roughly half of these organizations aren’t confident in their managed service’s people or technology. This is in comparison to those that leverage in-house SOCs, where confidence spikes to 75%.

So, we have a significant number of organizations who rely on managed services to protect them but are not confident in their abilities. What can be done to increase the confidence in their security?

At MITRE Engenuity, we believe that bringing threat-informed evaluation methodologies together with freely available results will drive both capabilities (people and technology) to improve their ability to detect and contextualize the threat, as well as improved confidence that these capabilities are protecting the organizations that leverage them.

Making a Technology Focused Evaluation More Personable

Over the course of the last four years, we have been working closely with technology providers to understand how they address known adversary behavior. We do so in a collaborative environment, so the vendors can speak for their own capabilities, as well as learn about threats and how to improve in their ability to defend against threats in the process. We have seen these capabilities improve over the years — what data is collected, how they use that data, and how they present that information to end-users. That said, in the previously mentioned survey, preliminary results have shown that people (training and hiring), not technology, is their main limitation, which goes directly to the point. Many organizations are looking for managed services to provide them the people to help.

In our original APT3 (2018) Enterprise evaluations, there were multiple vendors who included detections made by their services that they viewed as pivotal to describe their true capabilities. These results were treated as any other detection, though with a “delayed” modifier. In the next round of our evaluations, the APT29 (2020) Enterprise evaluations, we attempted to accommodate a growing services market and clearly delineate the uniqueness of that type of detection, by including an “MSSP” detection type in our evaluations.

In both the APT3 and APT29 evaluations, there was discussion by the end users on how to relate an MSSP detection to detections generated from the product. The main cause for this was that our methodology by design is open book, meaning we tell the vendor what we did. While this creates a great atmosphere for understanding and advancing technology, asking services to provide us context to what we did, which we have already provided them, does not offer the same value to end users nor to the vendors themselves. As a result, we decided not to evaluate services in either the Carbanak and Fin7 (2021) or Wizard Spider and Sandworm (2022) evaluations, despite continued vendor interest.

Both the interest and the data made the decision clear; ATT&CK Evaluations had to develop a new methodology that would allow end users to better understand how services address adversary behavior while also staying true to the core ATT&CK Evaluation tenants of transparency and collaboration.

Closing the Book, at Least to Start

One of the more unique features to ATT&CK Evaluations for Enterprise is that we conduct them via open book. What this means is we tell the vendor what we do, how we did it, and will even help direct them to the correct data to ensure we are not evaluating the defenders that the vendor brings to the evaluation, but instead the capabilities of their products.

For Managed Services, we are closing the book. No longer will vendors know what adversary we are emulating, nor will they know the Technique Scope, which defines what techniques could be included in our evaluation. Instead, the Managed Services will focus squarely on the defenders, and ask them to tell us what we did.

In an adversary emulation, similar to what you have become familiar with in our other ATT&CK Evaluations, we will conduct our attack in an environment that has been sensored by the participating vendor. The sensors that are leveraged will be the participant’s choice, though will be disclosed on our site. We will bring forward the detection-only rules of our Enterprise evaluations, meaning no protections or remediations can be performed that would prevent successful execution of the entire scenario. While this means that we will not evaluate every important aspect of services, it ensures we can explore the depth in service detection capability in an apples-to-apples way. We will continue to explore methodology extensions to allow us to evaluate protection and remediation capabilities of services, but for now, we have decided to focus on the visibility that services provide, which would allow a victim organization to understand what exactly happened in their environment.

Participants will deliver their analysis in a standard form factor that would be represented traditionally in a Service Level Agreement. The amount of context and supporting evidence, the form factor (e.g., dashboard alerts, reports, emails, etc.), will be as much a representation of “normal” as possible. Once the emulation is complete, MITRE Engenuity will map all delivered content to our undisclosed emulation plan.

A key aspect of this reporting is to note that vendors are not expected to detect all techniques. As we often state, not every technique is necessary to understand the intent of the adversary. It might help provide context, but in many cases might not be necessary to effectively describe the impact of an intrusion. A challenge, but also a goal, of these evaluations is to ensure we can assess a vendor who accurately summarizes essential behavior at a high level fairly to a vendor who enumerates every technique with supporting evidence. What is right for one organization might not be right for another, and our results must be able to support the variety of use cases.

Postmortem Purple Team

Collaboration is a key element of any ATT&CK Evaluation. We want vendors to learn and improve, and want them to walk away feeling they know exactly what we did and how they were assessed. In our Enterprise evaluations, we do this by walking through what we do, and hunt for the associated detections in real-time. In the Managed Services evaluations, we will still have a collaborative session, though it will be after the evaluation has ended, and the MITRE Engenuity team has mapped all provided information to the emulation plan.

During the collaborative session, MITRE Engenuity will announce which adversary we emulated and walk through the emulation step by step. At each step, we will describe what the red team did, mapping the actions to the noteworthy technique(s). We will also show any information the vendor provided that we believe accurately mapped to that behavior. During this session, the vendor is encouraged to ask questions, to understand what we did as well as ensure we understood the intent of the information they delivered. We will publish all information provided in the raw form, as well as the final mapping to the associated emulated adversary behavior, as determined by MITRE Engenuity.

Join Now!

We look forward to this expansion of the ATT&CK Evaluations program. If you are interested in participating in this inaugural Managed Service evaluation, please contact the team. If you leverage a managed service, ask whether they have considered being part of this evaluation, and we hope that, through this process, you can be more confident in your ability to defend against today’s threats. The call for participation closes February 25, 2022. We intend to run the evaluations in Q2, with results published in Q3.

© 2021 MITRE Engenuity. Approved for Public Release. Document number AT0022.