CyCraft AIR Achieves Analytic Detection Coverage in All 20 Major Attack Steps in MITRE Engenuity’s Latest ATT&CK Evaluations

The CyCraft AIR AI platform achieved 96.15% Signal-to-Noise Ratio with Zero Configuration Changes and Zero Delayed Detections Straight Out-Of-The-Box

CyCraft Technology Corp
May 3 · 9 min read

Taipei, Taiwan — 3 May 2021 — Results from the third round of the MITRE Engenuity ATT&CKⓇ Evaluations prove again that CyCraft customers are protected and secure from the most sophisticated real-world advanced persistent threats. The CyCraft AIR platform provided precise, actionable intelligence on each of the 20 steps of the evaluation with zero delayed detections and zero configuration changes to its detection logic, platform display, or endpoint sensors.

CyCraft AIR was pitted against emulations of two highly sophisticated adversaries of the financial and hospitality sector — Carbanak and FIN7. MITRE Engenuity monitored and documented CyCraft AIR’s performance against the two simulated cyber attacks through 20 major attack steps — broken further down to 174 substeps — on a Linux and Windows environment solely defended by CyCraft AIR.

CyCraft’s ATT&CK Evaluation results demonstrate how the CyCraft AIR platform uniquely secures enterprises via its dual cadence of patent-pending continuous automated forensic analyses, eliminating alert fatigue and providing precise clarity of visibility into an enterprise’s entire cyber situation.

Enterprises can benefit from CyCraft AIR through its many achievements in this ATT&CK evaluation round, including:

  • CyCraft AIR provided Analytic Detection (the highest detection classification) coverage through the entire attack by providing precise, actionable intelligence in each of the 20 major attack steps of the evaluation. CyCraft AIR provided enriched Analytic Coverage for each major step, not just raw telemetric data.

Out-of-Box Detections, Zero Configuration Changes & Zero Delays

Real-world environments generate massive amounts of noise — legitimate user behavior producing telemetry data; this, combined with innumerable alerts, false positives, and manual alert validation, constantly drains limited SOC and IT resources.

Two core issues to consider when mapping ATT&CK Evaluation results — which occurred in a clean environment — to the grittiness of a real-world environment are configuration changes and the quality of detections.

Here is a chart ranking all twenty-nine vendors by the total number of configuration changes and delayed detections across all 174 attack substeps. In this chart, the fewer configuration changes and delayed detections, the better.

CyCraft AIR required ZERO configuration changes while providing ZERO delayed detections.

CyCraft is one of the very few vendors who came prepared with a reliably and consistently fast, out-of-box solution requiring zero configuration changes.

What exactly are configuration changes?

After the initial execution of the attack scenario is completed, vendors have the opportunity to alter their solution in one of three ways: detection logic, UX, and data sources. This means that 24 out of the 29 vendors manually altered the investigation flow of their solution (detection logic), altered the display of data that was already collected but in its original state did not make it visible to the user (UX), or altered the sensor to collect new information (data sources).

Vendors can opt to perform configuration changes after the initial execution of attack scenarios. Cybersecurity solution vendors could have performed a configuration change for a multitude of reasons, including their tools initially failing to detect the attack substep. While configuration changes do occur in environments over time to advance capabilities, they provide a substantial hurdle to quickly investigating and responding to an attack.

Tools that don’t detect out-of-the-box means manual configuration, which counters the idea of automating the detection and response process in an already noisy real-world environment.

From day one of its inception, CyCraft AIR was designed as an out-of-box solution ready to detect and respond in any environment, therefore requiring zero configuration changes.

It is advisable to consider vendors that did not make configuration changes during the evaluation, or when considering a vendor who did make configuration changes, learning what those were, where they occurred in the attack, how those configuration changes altered their results, and what would need to be known and at which stages of an attack to implement the UX, data source, or detection logic configuration change to gain access to a potential detection as one step in a manual investigation.

  • 9% of all the detections made by all 29 vendors were due to configuration changes.

Only five vendors had zero configuration changes to their solution. Here is a chart ranking vendors by the total number of configuration changes made — here, lower is better.

CyCraft AIR provides true out-of-box detection — ZERO configuration changes required.

CyCraft is among the few vendors that came prepared with an out-the-box solution requiring zero configuration changes to detection logic, UX, or data collection via endpoint sensor.

Delayed detections, defined as detections that were not immediately available to the analyst, also represent a crucial component in investigations: If the detection is delayed, then that upsets the investigative workflow for analysts, slowing investigation and thus response times, leaving defenders more at risk.

Twenty-one of the twenty-nine vendors had delayed detections. Here is a chart ranking vendors by the total number of delayed detections made — here, the fewer delayed detections, the better.

CyCraft AIR provides reliably and consistently fast detection and response — ZERO delayed detections.

CyCraft is among the few vendors that came prepared with a reliably fast solution that provided zero delayed detections.

Clearest Signal

The MITRE ATT&CK evaluations took place in a clean environment where vendors have a good idea of what tactics and techniques attackers will use. In a real-world environment, vendors would not know that an attack was coming and would not know what tactics and techniques would be the focus of the attack — combined with the fact that in a real, live environment, defenders would have a lot of “noise” obfuscating the signal: legitimate user behavior producing telemetry data from the endpoints.

Several vendors take a “record everything” approach, which is the same reason why SIEMs have been seen as less than ideal in the detection and response context. When taking in so much data, much of it low value to forensic investigations, the investigation process is hampered: it is slowed down, and some things just get lost in the sea of “telemetry”.

A telemetry detection means that you would have to be actively seeking out that information in order to form a detection. Telemetry detections and the term “visibility” often cause a lot of confusion, because these two things essentially amount to raw logs, which can be a lot of noise to dig through as you will have telemetry on all the legitimate events and entities as well. An attack will often comprise less than 0.001% of your endpoint telemetry data. So If a vendor claims something like “complete visibility” defenders may find themselves digging through a mire of raw logs when using that product.

To shed more light on this, we analyzed the data provided by MITRE Engenuity and present the signal-to-noise ratio, which is taken as the percentage of detections sans configuration changes that are Analytic Detections. This shows how much actionable signal you can expect from a vendor.

CyCraft AIR automates detection and response without draining SOC resources on time-consuming noise or manual investigations.

CyCraft delivers the highest fidelity signal out of all the vendors because of the UX and forensic focus of our CyCraft AIR platform.

The UX of Cyberdefense

At CyCraft, we focus on the end user experience by delivering automated MDR reporting which enables defenders to completely understand an attack, visually and event-wise, and how to remediate it, without learning new terminology, querying, validating and triaging alerts, working on a workbench or the rest of what goes into typical SOC alert workflow and investigations, because we automatically perform the investigation for you with our continuous, automated, AI-driven DFIR platform.

With CyCraft AIR’s automated MDR reporting, we would have presented the attack in one report with information to remediate. Simpler than any other vendor. Zero querying. Automatic investigations.

The Logic Behind the CyCraft AIR Platform

CyCraft AIR was designed and built from day one to provide enterprises with precise visibility across all the endpoints in a single environment and beyond. CyCraft AIR achieves this by uniquely analyzing continuously collected intelligence via dual cadence of patent-pending continuous automated forensic analyses, which first monitors behavior on the endpoint level and then on a full-site, cross-endpoint level. CyCraft AIR’s novel dual-cadence system allows correlated alert triage automation in the context of the entire environment and attack story line, eliminating time wasted on alert validation and triage.

The reasoning behind the dual cadence system is that often can be the case where something that appears higher priority when viewed in an isolated context, is actually lower priority when viewed in a larger context (e.g., a new, unidentified program running on a host turning out to be a game). Conversely, an alert for something appearing to be a lower priority in an isolated context should be of higher priority when placed in a greater context (e.g., malicious file transfers). CyCraft AIR’s dual cadence DFIR system automates this commonly seen investigative context switching and linking, putting effectiveness back in the hands of defenders, as opposed to other tools, which are manual, time-consuming, and error-prone.

CyCraft AIR goes beyond detection and response dashboards and workbenches by providing automated incident response reports, highlighting critical priority threats automatically triaged while also providing precise actionable step-by-step intelligence leading defenders from response to elimination, remediation, and finally resilience, reducing the risk of similar attacks.

Everything Starts From Security

CyCraft customers can prevent cyber intrusions from escalating into business-altering incidents. From endpoint to network, from investigation to blocking, from in-house to cloud, CyCraft AIR covers all aspects required to provide small, medium, and large organizations with the proactive, intelligent, and adaptable security solutions needed to defend from all manner of modern security threats with real-time protection and visibility across the organization.

Engage with CyCraft

Blog | LinkedIn | Twitter | Facebook | CyCraft

CyCraft secures government agencies, police and defense organizations, Fortune Global 500 firms, top banks and financial institutions, critical infrastructure, airlines, telecommunications, hi-tech firms, SMEs, and more by being Fast / Accurate / Simple / Thorough.

CyCraft powers SOCs using innovative AI-driven technology to automate information security protection with built-in advanced managed detection and response (MDR), global cyber threat intelligence (CTI), smart threat intelligence gateway (TIG) and network detection and response (NDR), security operations center (SOC) operations software, auto-generated incident response (IR) reports, enterprise-wide Health Check (Compromise Assessment, CA), and Secure From Home services. Everything Starts From Security.

Meet your cyber defense needs in the 2020s by engaging with CyCraft at


Craft for Resilience

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store