Blue Team 201: Detection — Where Do You Start?
Many of you may have heard the old adage “prevention is ideal, but detection is a must.” For the detection piece in this blue team series I decided to focus on endpoints specifically. I know this does not cover everything, especially those of you who are responsible for protecting things you can’t just slap an AV or agent on (sorry ICS folks). I also find endpoint detection to be more straight-forward than network detection if you are starting from scratch. If you remember the pyramid model from my last post, two of the bottom tiers asked the questions: “Do you have visibility across your assets?” and “Can you detect unauthorized activity?” Detection can be analogous to playing a game of “find the needle in the haystack”-although sometimes it feels as if its a game of “find the needle in the needle-stack”. With that said, lets start with the basics and get down to it.
What do we want to accomplish here? I don’t know about you but ultimately I would like my blue team to be the Eye of Sauron. We want to see everything happening in our environment so we are able to detect any activity, malicious or not. We know the logs do not lie, so what are the first things we can start auditing to acquire visibility in our environment? For the purposes of this post I am going to assume we’re in a primarily Windows Enterprise environment to keep things simple.
One of my favorite resources of all time is the logging cheat sheet by Malware Archaeology. This guide is an utterly fantastic starting point to determine what kinds of events you want to capture in your Windows domain and why. To determine what you want to enable, ask yourself questions like: Can I see every time a user logs on or off a machine? When they change their password? Or even when they start a new process? Not all of this information is available by default so we’ll have to turn some stuff on. A great way to accomplish this and manage it is through group policy. I found myself repeatedly looking up which group policy settings correspond to which Windows event codes, so here is all that information in one place to complement your windows logging cheat sheet. If you’re not yet familiar with windows event logs and the context they bring to infosec, here are a few places to start: Peeping Through Windows Logs, Spotting the Adversary with Splunk, Sexy Six Event Codes.
Once you are confident in your visibility, the next step is to get all those logs into one place so your team can easily access them for investigation. There are many ways to accomplish this and therefore I will try my best to be vendor-agnostic in this series. With that said, several of the resources I mention here use Splunk, but almost all of the insights presented can be used in other platforms. There are countless alternatives out there like AlienVault USM, IBM Qradar, GrayLog, ELK, LogRhythm, etc. You also don’t have to use the Windows native event log as EDR agents have become so popular nowadays and gather much of the same telemetry if not more.
Like I mentioned earlier, I will be talking about detection around the endpoint, but obviously that is only one piece among the diverse IT infrastructure you may have in place. We will not forget about those firewalls, proxy servers, VPN, DNS, and other network device logs. In another post I will explore the field of network security monitoring and all the detection powers used by IDS/IPS solutions such as Zeek (Bro), Snort, Suricata, etc. But if you’re too eager to wait I highly recommend picking up a copy of Practice of Network Security Monitoring. With that in mind, I firmly believe that when you are starting out, the most manageable and pragmatic place to start is the endpoint, as it is the most conclusive place to look. Once you have enabled logging for windows events and start to become familiar with what is normal in your environment, now the real fun begins.
Windows event logs are super valuable when it comes to detecting malicious activity but it won’t catch everything. So can we take it a step further? Sysmon is a free and much more granular event logger that is a part of Mark Russinovich’s Sysinternals tool suite for Windows. What makes sysmon so valuable is that its a kernel-mode driver, which for our purposes means it can provide even greater visibility into the operating system. SwiftonSecurity’s sysmon configuration file is a great place to start as this type of logging has the potential to be very noisy (Malware Archaeology has a cheat sheet for sysmon as well). Not all organizations have the budget for enterprise security products and this is why I am such a huge advocate for FOSS security solutions. I consider sysmon to be so valuable when it comes to detection because it is free and just as good, if not better than many commercial EDR products.
Now that you have visibility into all your endpoints, you can focus on improving your detection. There is a multitude of malicious “things” out there you can detect and it can be very overwhelming to determine which ones you should care about and why. So if you’re not sure how to measure the maturity of your detection capabilities, refer back to the cheat sheets I mentioned earlier. At this point you may have some out-of-the-box alerts from any security tools you have or maybe wrote/curated some alerts yourself. False positives are forever the chronic affliction of the blue team and now we can work towards improving the fidelity and durability of our detection. A great illustration of this is the model SpecterOps has coined the Funnel of Fidelity. This is the process we want to use to help finding that needle in the haystack less difficult. Just keep in mind for every alert, notification or email that reaches your inbox, would you rather it be right 9/10 times or 1/10 times?
Having said that, I hope I’ve provided you with some great resources to start building your detection stack and turn your blue team into the omnipresent Eye of Sauron. If you want to take this a step further I encourage you to look at Event Tracing for Windows (ETW) and see how in-depth you really can get with the operating system logs. I also recommend checking out the Wazuh project as well as Security Onion for those of you who are interested in free and open-source detection technologies.
And remember, think back to the IR pyramid mentioned in my previous blog- asset inventory & management comes before telemetry & visibility in the hierarchy. Here is a scenario to paint the picture: A user reports opening a phishy attachment to the servicedesk, you go to check the logs and uh-oh, there’s nothing there because the EDR agent wasn’t installed or windows events were not forwarded to your SIEM. It can happen. Be patient in this process as telemetry is the building block of detection. Once you have all the data there, you can really get your hands dirty and sharpen your blue team skills. This stuff gets me pretty jacked up, so if you had some fun with this- grab a red bull and stick around for the next post where I will explore more advanced blue team tactics like detection engineering, catching windows tradecraft and threat hunting. Remember: the bad guys only have to get it right once, but we have to get it right every time. I hope this post provided a good introduction to the art of detection, so when someone makes a sound- you hear it.
Some more resources on detection below:
Splunkmon-Taking Sysmon to the Next Level
Tracking Lateral Movement Part One
Hunting for Credential Dumping in Windows Environment
SANS:Detecting Security Incidents Using Windows Workstation Event Logs
Detecting Penetration Testers on a Windows Network with Splunk
LimaCharlie (not exactly free but pretty cool pricing model)
