Jean-Claude CoteinTowards Data ScienceOptimizing Sigma Rules in Spark with the Aho-Corasick AlgorithmExtending Spark for improved performance in handling multiple search terms8 min read·3 days ago----
Jean-Claude CoteinTowards Data SciencePerformance Insights from Sigma Rule Detections in Spark StreamingUtilizing Sigma rules for anomaly detection in cybersecurity logs: A study on performance optimization14 min read·Jun 1, 2024----
Jean-Claude CoteinTowards Data SciencePerformant IPv4 Range Spark JoinsA Practical guide to optimizing non-equi joins in Spark9 min read·Jan 25, 2024--1--1
Jean-Claude CoteinTowards Data ScienceUnleashing the Power of SQL Analytical Window Functions: A Deep Dive into Fusing IPv4 BlocksHow to summarize a geolocation table by merging contiguous network IPv4 blocks8 min read·Jan 10, 2024----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection Using Sigma Rules: Build Your Own Spark Streaming DetectionsEasily deploy Sigma rules in Spark streaming pipelines: a future-proof solution supporting the upcoming Sigma 2 specification13 min read·Jun 12, 2023----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection using Sigma Rules (Part 5) Flux Capacitor OptimizationTo boost performance, we implement a forgetful bloom filter and a custom Spark state store provider8 min read·Mar 17, 2023----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection using Sigma Rules (Part 4): Flux Capacitor DesignWe implement a Spark structured streaming stateful mapping function to handle temporal proximity correlations in cyber security logs6 min read·Mar 2, 2023----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection using Sigma Rules (Part 3) Temporal Correlation Using Bloom FiltersCan a custom tailor made stateful mapping function based on bloom filters outperform the generic Spark stream-stream join?6 min read·Feb 14, 2023----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection using Sigma Rules (Part 2) Spark Stream-Stream JoinA class of Sigma rules detect temporal correlations. We evaluate the scalability of Spark’s stateful symmetric stream-stream join to…7 min read·Feb 2, 2023----
Jean-Claude CoteinTowards Data ScienceAnomaly Detection using Sigma Rules (Part 1): Leveraging Spark SQL StreamingSigma rules are used to detect anomalies in cyber security logs. We use Spark structured streaming to evaluate Sigma rules at scale.8 min read·Jan 24, 2023----