Why does EDR matter?

Laptops and PCs are an important point of entry for cyber attackers because they are operated by humans that can be easily tricked into clicking on malicious links or email attachments. When those laptops and PCs are used in an enterprise, they are called endpoints and often run special software to mitigate cyber threats.

Which EDR performs best?

Thanks to MITRE, a US non-profit organization, we can now compare the performance of various Endpoint Detection and Response solutions. This evaluation is unique because it puts a well-documented cyber threat in a lab environment and tracks detection throughout the attack path. MITRE published the results but deliberately without ranking, scoring or rating. Make up your own mind.

This is MITRE. They solve problems, and apparently, they're not afraid to point out that "cyber" is the fifth domain in warfare besides land, sea, air, and space.

MITRE solves problems
Figure 1. MITRE solves problems.

In the cyber domain they’re famous for creating the MITRE ATT&CK matrix, an information product that helps organizations think about their cyber defense in a more attacker-oriented way: from initial access via privilege escalation and lateral movement to impact.

We approve of us!

MITRE doesn't assign scores in their EDR evaluation, and in this ranking vacuum, you can only imagine what most vendors did:

  • "Carbon Black outperforms all other EDR solutions" (source)
  • "CounterTack Platform leads with fast automated detections" (source)
  • "CrowdStrike Falcon […] the most effective EDR solution" (source)
  • "[…] Endgame as the first zero training endpoint protection" (source)
  • "FireEye Endpoint Security […] the most effective EDR solution" (source)
  • "[…] evaluation showcases the effectiveness of SentinelOne’s platform" (source)
  • "[…] Windows Defender ATP demonstrated industry-leading optics […]" (source)
  • "[…] Cybereason best enables defenders […]" (source)
  • "Cortex XDR and Traps Outperform in MITRE Evaluation" (source)

What are the results?

Loading the evaluation results into Splunk, via this Python script, leads to the charts below.

The evaluation simulated 136 steps of an advanced persistent threat. For example the first chart shows that 60 attacker steps on a total of 136 weren’t detected by the chart leader of main detection type "None". You can read up on the main detection types here.

Who detected the least APT3 steps?

It may not be a coincidence that RSA is missing from the cheering press release crowd.

Detection fails
Figure 2. Detection fails.

Which EDR system logs most telemetry?

If you just want the logs and create your own rules in a SIEM, look no further than this chart (or Sysmon).

Who logs the most events?
Figure 3. Who logs most?

Who enriches events the most?

Enrichment is telemetry++.

Who enriches events the most?
Figure 4. Who enriches events the most?

Whose IOC triggered on APT3?

Whose IOC triggered on APT3?
Figure 5. Whose IOC triggered on APT3?

Who detects most general attacker behaviour?

Who detects most general attacker behaviour?
Figure 6. Who detects most general attacker behaviour?

Who detects most specific attacker behaviour?

Who detects most specific attacker behaviour?
Figure 7. Who detects most specific attacker behaviour?

Another dimension

Three other modifiers are included in the EDR evaluation. Read up on them here.

Tainted

Tainted means correlated, which is helpful for analysts.

Tainted results
Figure 8. Tainted results.

Delayed

This means non-realtime.

Delayed results
Figure 9. Delayed results.

Configuration change

This means APT3 steps were detected, but only after special vendor assistance.

Configuration change results
Figure 10. Configuration change results.

Which EDR is most suitable as ultimate noise generator?

If you only care about raw data and do correlation on your own outside of the EDR solution, this chart ranks vendors based on Telemetry, Enrichment and IOC results:

Ultimate noise generator
Figure 11. Which EDR is most suitable as ultimate noise generator?

Which EDR has the best signal/noise ratio?

If you look for EDR solutions based on signal/noise ratio, this chart ranks vendors based on General or Specific Behaviour divided by the amount of Telemetry, Enrichment or IOC results:

Best signal to noise ratio
Figure 12. Which EDR has the best signal/noise ratio?

Draw your own conclusions

There are endless possibilities to slice and dice the EDR evaluation results. Drop me a line with how and why if you did.

  1. Download Splunk for free.
  2. Download and install the EDR evals app from Splunkbase.