The importance of reference data for effective threat detection

Reference data, also known as Enrichment data, is a crucial component of enabling accurate threat detection rules.

For example there is zero value in creating a rule when a shell is spawned on an engineers desktop. But what about if it was for the CEO’s assistant?

What about a system that is constantly scanning for vulnerabilities in your network? At first glance sounds completely malicious… until you realise it is the vulnerability scanner.

By adding the context of the system or user type you can begin to ask more insight questions of your log data. Without reference data you are shooting in the dark and raising issues that more likely than not have to be explained with people instead of being done as part of the analysis process itself.

Below are but some of the reasons that if you do not have a robust approach to the collection and use of your companies reference data you should!

1. Establishing Normal Behavior: Reference data helps establish a baseline of normal behavior for various entities within the organisation, such as users, hosts, applications, and network traffic. By understanding what is considered typical, the detection system can identify deviations or anomalies that may indicate potential security incidents.
2. Identifying Anomalies: Detection engineering relies on identifying anomalies that deviate from established norms. Reference data aids in defining what is “normal” and enables the detection system to raise alerts when activities fall outside those parameters.
3. Threat Intelligence Integration: Reference data often includes threat intelligence feeds, which provide information on known malicious actors, indicators of compromise (IOCs), and emerging threats. Integrating threat intelligence into the detection system enhances its ability to detect and block malicious activities.
4. Pattern Recognition: By analysing historical reference data, detection systems can identify patterns and trends that might indicate ongoing or recurring attacks. This knowledge helps improve the system’s ability to recognize new instances of known attack patterns.
5. Supporting Machine Learning Models: Machine learning models used in detection engineering require labeled datasets for training. Reference data that includes historical security incidents and their outcomes can serve as valuable training data for these models.
6. Tuning Detection Rules: Reference data aids in fine-tuning detection rules and algorithms. By analysing past incidents and their associated data, the team can adjust detection thresholds and parameters to optimise the detection process.
7. Post-Incident Analysis: After a security incident, reference data provides critical information for post-incident analysis. It helps understand the scope, impact, and techniques used in the attack, allowing the organisation to strengthen its defenses against similar future threats.
8. Benchmarking Performance: Regularly comparing detection performance against historical reference data enables the team to measure the efficacy of their detection capabilities. This helps identify areas for improvement and ensures the system stays effective against evolving threats.
9. Compliance and Reporting: For regulatory compliance and reporting purposes, reference data can serve as evidence of the organisations proactive security measures and the effectiveness of its detection and response efforts.
10. Situational Awareness: Having access to comprehensive reference data provides the Detection Engineering team with a broader understanding of the organisations security posture. This situational awareness enables better decision-making and strategic planning for improving cybersecurity defences.

Reference data is but one of the many steps we go through with our customers at Illuminate Security when onboarding to ensure that the detections achieved are of high quality, accuracy and applicability to their environments.

Sign up today and find out what effective threat detection can mean for your organisation!