The Network Logs You Need
How can you tell if your host is connecting to a C2 server?
If you still think you only need identity to secure your environment and no network security controls, you probably haven't studied many cyber attacks. Please start with this series of blog posts on the Solar Winds Hack. Understanding C2 servers is one of the most important concepts you need to know in cybersecurity.
In my last few posts, I explained some concepts that might help you monitor network traffic, starting with this basic explanation.
Then I explained that there is one simple rule that can block a lot of network traffic. If you followed the instructions closely you will notice that I recommended blocking traffic on the WAN or Wide Area Network interface only on my firewall for incoming traffic from the Internet. If the connection is initiated from the Internet to your machine, it is likely someone scanning the Internet for open ports, a typo when someone typed an IP address or configured a machine, or some other questionable behavior.
One Rule To Identify Network Noise
One basic rule filters out a whole lot of noise on your network
Most well-designed client-side Internet-connected products and software do not initiate traffic from the Internet to your device, allowing you to safely ignore this traffic. End-user or client devices may be things like your Laptop, a wifi router, or your Alexa. The type of systems that allow incoming traffic are things like web servers, DNS servers, load balancers, and systems hosting application programming interfaces (APIs). These systems are designed to receive requests from many sources and provide a response with the requested information.
In fact, I suggested that home network users might even want to not log that traffic at all. Security people who usually want every log they can get their hands on may have freaked out a little bit when I wrote that. I’m not talking about production systems. I’m only talking about pure, useless, meaningless noise from the Internet that gets blocked anyway and is never going to be valid or useful in any case, presuming you aren’t using network protocols that break traditional Internet architecture. It’s pure and simple noise.
Someone scanned your network. Ok if you’re going to let them keep accessing things on your network you want to know everything they did. If you’re just going to completely block all the activity from that network and you are sure your firewall is working it’s just a log full of blocked packets. You can use that logging and processing power for something more interesting and potentially stave off a DDOS attack.
The Network Logs You Might Not Need (GASP!)
Traffic you might not want to log in certain cases and why
OK so you’ve blocked that traffic initiated from the outside in, and you know all those IP addresses are generally doing something they shouldn’t be doing. None of it is related to the functionality of devices on your network and the traffic that you want or need. Those IP addresses seem to be 100% up to no good.
There’s another set of rules you might want to add. You may want to know when the devices on your network try to connect to one of those questionable networks. Adding these rules can help you for multiple reasons.
First of all, you may have inadvertently blocked something that a person or device on your network needs to access. Network security professionals with significant others know what I’m talking about! I’ve heard other network geeks like myself talk about this in presentations.
Just the other day someone in my household couldn’t get to his fantasy football league. Oops. This was unrelated to these questionable networks, but by taking a look at what got reject from the particular internal interface (LAN) his device was using, I could find the blocked traffic and fix the rule.
The other thing that creating an outbound rule showing whenever one of your devices connects to one of these seemingly sketchy networks is that you can see potentially malicious activity. If you know the network is showing up in your logs scanning for vulnerabilities, chances are, other nefarious activity may be coming from that network.
If you visit a website, for example, they may be selling ads that are hosted on one of these networks and those ads may contain malware. When you visit the website, it’s serving you legitimate news, but it may also be delivering something unwanted in the many third-party connections your system makes when you visit that site. I explained this concept in relation to the number of connections your system makes to third-party servers when you load the CNN website in this blog post.
Real-time third-party code injection
The risk associated with inclusion of externally-hosted code on websites and how to mitigate it
It could also be that the network hosts a combination of innocent and malicious user activity. For example, I know some legitimate websites get hosted on Digital Ocean. I don’t know if the business owners realize what else is coming from that network. I saw a huge uptick in traffic from Digital Ocean today scanning my network. But I’ve also had to allow traffic to a website (rarely) hosted on Digital Ocean.
For these reasons you want to see when the systems on your network are trying to connect to the nefarious networks that you otherwise ignore when they are sending traffic to you. You want to set up a rule on your LAN interface and whatever other internal network interfaces you have on your firewall. This rule should track whenever the source is your LAN network and the destination is one of the networks sending unwanted traffic. You want to LOG that traffic.
Also note that I never add OUTBOUND traffic to a rule that does not log that traffic. I always log outbound traffic no matter what (where the source is not an external IP address). After I wrote this article, some weird traffic appeared to an AWS EC2 instance on port 8080 from my network. That traffic was particularly disturbing since it only appeared on the WAN going outbound from the external ISP IP address. No corresponding traffic appeared on the local network. You need to know what outbound traffic exist on your network whether it’s coming from the WAN, LAN, or some other interface. I posted the IP address and port to Twitter and miraculously the traffic stopped. Not sure if it was coincidental because I was also deleting channels from Roku that were having issues and we have a TV that I inherited and don't particular love how it operates on the network. Don’t have time to investigate further but I know that particular traffic got blocked and the moral of the story is: Log all outbound traffic. Still wondering what happened and will need to keep an eye on it.
It’s so ironic that I’m telling you just how important is to log outbound traffic since I had to argue with my hosting company to turn those logs on back when I experienced my first data breach. In fact, that experience and arguing with them to turn on those logs got me into cybersecurity. It also helped me discover what the attackers were doing on my system: sending spam on outbound port 25, among other things.
How network traffic got me into cybersecurity
Also — being paid by a large hosting company to go away after reporting a security incident, and other strange events.
This traffic is very important because it could indicate you have some malicious activity in your network. It could also just be a misconfiguration or some functionality you don’t understand. So once you see one of these log entries, the next step in the incident response process I describe in my book and video on getting a job in cybersecurity is to investigate. You want to figure out which device is connecting. You might want to use a packet sniffer on the device to see what is in the packets or capture the packets from a network device and view the pcap file.
In my case, I noticed this traffic in my logs to IP address 184.108.40.206.
At some point, I received traffic from this network that indicated a device on their network was attempting to initiate a connection to mine, indicating a possible scanner. I blocked the network. Later, as I’m checking my network logs, I see that something on my network is attempting to connect to this IP address.
I can use dig to try to reverse the DNS record to see if a domain is related. In this case, I get edge-488.bunnyinfra.net.
The problem with the above is that the domain name may be associated with a CDN. I wrote about that problem here:
In order to check to see if this IP address returns a website or there’s an SSL certificate associated with this IP, I have to open up the firewall to visit the address. If the address is serving up malware I don’t want to visit this from my own machine I want to do it in a cloud virtual machine with a tightly locked down network. I can inspect all the traffic and behavior on the cloud network logs when I visit the site to see if anything suspicious is going on.
I can visit http[:]//220.127.116.11 to see if the IP returns a web page.
I can visit https[:]//18.104.22.168 in a browser like Safari (Chrome doesn’t always work) to see if I get a failed SSL certificate error indicating the actual domain instead of a generic CDN domain if one exists.
Next, you can search for the IP address in multiple places to see if you can get more clues about it. Check AbuseDB. This IP has been flagged:
The report is 6 months old, so it could be that the IP address is no longer involved in malicious behavior. In any case, here’s the report.
Chunked encoding could be related to HTTP Request Smuggling, something we test for in 2nd Sight Lab penetration tests. This could lead down a rabbit hole to try to see what is going on with this particular IP address but not going to delve into it further right now. It could be that the owner of the application fixed the vulnerability that triggered the above report.
I can go further by searching Google for the company, the IP address, and the related domain names to see if I find anything else about this network in addition to deep inspection of the packets to figure out what is actually happening on my own network.
At the moment the traffic has subsided, so I’ll have to watch for it and see if it shows up again. If and when it does I can also try to figure out if this traffic shows up when I visit a particular website and then look at the site using browser developer tools to try to reverse engineer anything interesting.
If I had a piece of malware on a host connecting to this IP address, it might be a C2 or command and control server (explained in the book and video I mentioned above) sending malicious commands to my systems. That’s why I’ll want to determine exactly what the traffic is doing and what process on what device is causing it. It could be completely benign, but you don’t know either way until you perform the investigation. And in order to know what to investigate, you need the logs.
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2021
Check out the next post with some tricks for creating less firewall rules and less time analyzing noncontiguous networks you don’t need to access.
Concatenating IP Ranges And Other Firewall Rule Tricks
Tips for fewer firewall rules when you’re trying to block traffic from a network
Want to learn more about Cybersecurity and Cloud Security? Check out: Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
For a recap of cybersecurity news last week check out the 2nd Sight Lab Cybersecurity News Blog. Malware, vulnerabilities, data breaches, cost of a data breach, cybersecurity laws, and interesting cybersecurity developments.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts