Data Science & Exploratory Data Analysis: the Panda versus the Pony!
Exploratory data analysis sits at the core of any insightful data work. Performing log analysis in search for threats is no different.
Whether you are a security analyst triaging alerts or a detection engineer looking for detection opportunities from logs, the ability to understand data is what determines your success.
Now that security logs are abundant, speaking to data is what sets you apart from an average cybersecurity pro with domain knowledge 'only'.
What is EDA?
In statistics, exploratory data analysis (EDA) is an approach of analyzing data sets to summarize their main characteristics, often using statistical graphics and other data visualization methods — Wikipedia.
In the context usually covered here, mainly around detection engineering and security analytics, that's the process of digging into log data to answer the questions we often have when facing a new log data source (dataset).
Exploratory data analysis is the discovery of trends and patterns in data using statistics and visual representations.
Many individuals in our industry already do that without knowing, myself included (a few years back). EDA is the entire process of untangling a brand new log source to figure out what’s inside!
