MITRE Engenuity ATT&CKⓇ Evaluations Results from Deception Trials

Published in

MITRE-Engenuity

9 min readMay 25, 2022

Providers of non-detection-oriented security solutions need objective, transparent methodologies to evaluate and frame their value propositions. This need was the driver behind the launch of MITRE Engenuity ATT&CK® Evaluations Trials, an exploratory new program designed to help niche solutions providers realistically describe their strengths in defending against known adversary behavior.

Results from the first Trials project, which focused on Deception solutions, are now available and published on the ATT&CK Evaluations website. In keeping with our commitment to transparency and trust, here’s a behind-the-scenes look at the participants, methodology, and findings.

Participant Snapshot

Deception product participants in our Deception Trials project were Attivo Networks (recently acquired by SentinelOne) and CounterCraft Security. As two of the foremost providers of Deception capabilities, our two vendor participants represent a fantastic snapshot of unique Deception products that have inherently different approaches. Accordingly, each have different views on how their solutions fit into a larger security architecture. Please note, we are directly citing responses to vendor participants who we’ve interviewed. The views reflected in the quotes are those of the vendors, and not verified or tested by MITRE Engenuity.

SentinelOne’s VP of Engineering, Srikant Vissamsetti explains their organization’s perception of how a Deception product fits into a larger security infrastructure.

“Deception provides early detection of intruders with high fidelity detection and no false positives. Deception products misdirect intruders away from production assets and — study adversary behavior and motive with detailed forensics that helps in understanding the motivation of adversaries and the techniques and tools they use. The cloaking capability built into Deception products provides advanced protection by hiding credentials, Active Directory objects and data, and forces intruders into Deception systems. Deception, along with other security products like EDR and XDR, enhances and strengthens enterprise security.”

David Barroso, Founder and CEO of CounterCraft, further explains the importance of a Deceptions solution in any security infrastructure,

“Deception is the only way to provide advanced warning of how an organization is being targeted by their adversaries. It provides high-fidelity alerts on attack patterns, tactics, techniques, and procedures and is the only way to build a detailed profile of the who, what and why of an attack, as it relates directly to you. This detailed threat intelligence is timely, focused, and above all actionable because the intel being collected is directly related to your own organization and your attack surface.
Deception is the new kid on the block and CounterCraft has consciously designed our platform to fit seamlessly into an existing security infrastructure. With over 45 integrations to third-party security systems out of the box, fitting in with incumbent systems is simple, either to add valuable threat intel feeds to existing analysis systems or interacting with orchestration systems to provide automated cross-platform attack response.”

With that frame of reference in mind, the next section will explore the process of our Trial Evaluation and identify each vendors’ interest in participating.

Why Deception?

As we embarked on our journey to design a process to evaluate Deceptions solutions, we first had to ensure that there was a need in the market for a Deceptions Evaluation. As is the case with all our offerings, this Trial virtually caters to a two-sided “marketplace.” On the demand side, we have our end-user community, and on the supply side, we have our vendor ecosystem.

Our end-user community consists of security practitioners: the folks who defend our data and privacy. We believe this to be the most important stakeholder persona in our Evaluations and we prioritize surfacing objective results to be consumed by the public. In fact, our first core goal of Evaluations is focused on empowering end-users with objective insights into how to use specific commercial security capabilities to address known adversary behaviors. To that end, we decided that Deception capabilities were a great place to start with Trials because there weren’t any other evaluations that focused on Deceptions specifically. The need existed and we decided to open up the Call for Participation for this round.

On the other side of the “marketplace” lies our vendor ecosystem, which consists of the organizations who develop and support products that help our security practitioners run their security program. Like many other cybersecurity providers, these Deception vendors sought emulation and other forms of evaluation to better describe their solution’s value and inform its evolution.

Srikant further explains his organization’s interest in participating in our first Trials Evaluations,

“MITRE Engenuity ATT&CK® Evaluations has a reputation for performing thorough and transparent evaluations of security technologies over the years. The testing is systematic and comprehensive and tests solutions’ ability to protect against adversary behavior. SentinelOne supports in its effort to provide transparent and fair methodologies to evaluate adversary behavior.”

As is the case with the varied approach to Deceptions solutions themselves, David explains CounterCraft’s distinctive initial interest in joining Trials Evaluations,

“CounterCraft participated to compare our Deception tech with other vendors in a clear and neutral way. Standard minimal functionality and performance criteria do not exist in the Deception space. This is confusing for end users when evaluating Deception technology. We hoped these trials would change that. Standard testing criteria would benefit end users and allow different vendor approaches to shine in different ways. We see MITRE as the best placed organization to create this common and fair testing framework and we are proud to be able to invest in the Trials to provide value to the whole Deception user and vendor community.
The CounterCraft expectation was to collaborate with MITRE Engenuity in the creation of the testing methodology to fully define metrics that measures the different benefits of a Deception platform and help provide a way of identifying key strengths of each technology. The goal of this is to inform potential users and help them understand how best to deploy Deception according to their needs. This expectation has not been reached yet, but this initial Trial is a start and provides many areas to build on.
At the forefront of our minds was the possibility of providing an open and impartial analysis of Deception technology, both to raise awareness and to showcase the CounterCraft approach to Deception deployment.”

Deception Evaluation Process

The MITRE ATT&CK Evaluations team were challenged to design a methodology that would fairly evaluate the varied, unique range of Deception-based defenses while also articulating key differences in products and strategy.

Ultimately, the APT29 Enterprise Evaluation framework was updated, revised, and applied. This emulation is based on the APT29 adversary, distinguished by its stealth, sophistication, arsenal of custom malware, and dynamic operational cadences influenced by perceived intelligence value and/or infection method of victims. The evaluation itself followed a two-phase process:

● Phase 1 — Observe: Will the adversary encounter the Deception? During this first phase of the evaluation, emulations were run with the goal of capturing what each Deception solution deployed when faced with adversary behavior. Phase 1 results offer a valuable indicator of the degree and circumstances in which each Deception is capable of impacting the adversary.

● Phase 2 — Engage: Does the Deception affect the adversary? During the second phase of the evaluation, emulations were repeated. This time, the adversary technique engaged with the Deception, and the outcomes of those interactions were recorded. Phase 2 data reveals more about how each Deception performs against a variety of adversarial behaviors.

Deception Evaluation Results

Deception products are not as straightforward to evaluate as detection-oriented security systems, and they are far more nuanced than detection products. Before comparing results and weighing participants’ solutions against one another, note again that the two products have fundamentally different approaches.

Srikant describes some of the key Deception capabilities within the SentinelOne suite.

“SentinelOne delivers comprehensive detection coverage with Deception authenticity and scalability across all attack surfaces (on-premises, cloud, and remote sites). Additionally, it goes beyond decoys and breadcrumbs by adding concealment and misdirections to its offering. SentinelOne uniquely creates decoys for everything from Windows endpoints to medical devices and energy substations.
SentinelOne protects against Active Directory enumeration and ransomware attacks. It hides files, AD objects, folders, mapped network and cloud shares, and removable drives so intruders can’t find the data or access it to steal or encrypt. SentinelOne gathers and automatically correlates data from the attack, including memory forensics. Extensive third-party integrations and playbooks provide automated incident response and negate the need for additional resources to analyze and respond to an incident.”

On the other hand, CounterCraft has a different approach, which David dives into.

“The CounterCraft approach to Deception deployment and the core of our technology is to provide high-fidelity, actionable threat intelligence in real-time. Our fully instrumented Deception environments and stealth communication channels allow monitoring of attacker behaviour, detecting activity, providing enriched threat intel, and allowing the defender to manage the attack and protect their assets.”

Assessing these different approaches is extremely valuable as you determine how to read through the results and determine which solution may fit your security program. For those of you who are familiar with the Enterprise Evaluation results, keep in mind that we do not have Detection Categories for these results. Instead, we created Deception Categories that are based on the terminology found in the MITRE Engage framework. For these reasons, It is even more important than usual to assess the gaps you are trying to fill before you look to examine the results.

Full results are available on the ATT&CK Evalutations website. Note that we do not assign scores, rankings, or ratings; instead we offer unbiased evaluation results so that other organizations may provide their own analysis and interpretation. Learn how to best leverage and interpret ATT&CK Evaluations results.

Deception Trials Program Takeaway

One intention behind the Trials program is to provide fresh insights that will contribute to the ongoing innovation and evolution of threat-informed defense. From a research point of view, this project revealed the complexity involved with comparing inherently different Deceptions products. It’s therefore unlikely that Deception evaluations will be reliable at scale any time soon.

We hope that these Evaluations help bring the vendor ecosystem forward in their prioritization of features and functionality for addressing known adversary behavior. It is for this reason we are looking to diversify our overall Evaluations offering with inclusions of both products and services. Accordingly, we hope that the vendors who participate reap the benefits of collaborating with us on these research projects.

Srikant explains SentinelOne’s intent on participating in this round,

“SentinelOne is an enthusiastic supporter of what MITRE does, bringing transparent and open evaluation methodologies to the security industry to help define and continually expand a common cybersecurity language that describes how adversaries operate.
SeninelOne had tremendous success with the ATT&CK Enterprise Evaluations, and participating in all the evaluations has become an essential practice that we have used to improve our products further. This also provides SentinelOne an opportunity to participate and pioneer the inaugural ATT&CK Deception Trials and showcase the platform’s security diversity and depth.
SentinelOne Deception products helped hundreds of customers in preventing privilege escalation, lateral movement, and collecting threat intelligence using deception and identity cloaking technologies. While many customers benefitted from SentinelOne technology, it was never documented publicly for others to see the benefits of deploying deception products.
Customers understand that MITRE [Engenuity] has a very stringent testing methodology, and they can trust the results more than any claims made by a vendor. It is also challenging for customers for themselves to perform comprehensive evaluations as MITRE [Engenuity] does. We believe the results will accelerate the deployment of deception technology and let customers select the right vendor to work with.”

David further describes a desire to collaborate on further work to improve the testing methodology,

“CounterCraft expectation was that the testing would be a collaborative process allowing all deception vendors who participated to define the testing strategy based on our experience combined with the experience of MITRE in the deception field. The testing process still requires work to achieve this, but we are keen to further our collaboration with any future work.
CounterCraft hopes that in the future the evaluation process will provide an independent analysis, on common ground from a known impartial source, to help identify how best to deploy deception. It should provide a benchmark to measure all deception platforms equally and highlight strong areas and help identify the consumers of the results which is the best tool for their specific deception deployment.”

The takeaway for organizations comparing Deception solutions is to appraise each one on the basis of its unique use case or scope. Subjective analysis is required to determine whether any Deception solution is the best addition to the diversity and depth of current cyber protections.

More from ATT&CK Evaluations

Thank you to the vendors who participated in the initial round of our ATT&CK Evaluation Trials program, and to the collaborating MITRE Engenuity teams working to advance the state of threat-informed defense.

The MITRE Engenuity ATT&CK Evaluations program is on a mission to make a safer world with a threat-informed defense approach to security. To learn more or get involved, visit the ATT&CK Evaluations website. For more about MITRE Engenuity, a tech foundation for public good, follow us on Medium, LinkedIn, or Twitter.