Threat Hunting and Detection with Web Proxy Logs

Let’s Connect | LinkedIn | Twitter

image from pixabay

As I mentioned in my previous post about detecting and responding to ransomware attacks, I created a hunting and detection guide using web proxy logs.

Web Proxies generate a common set of information that can be used for threat hunting and detection. This information contains Duration, HTTP Status, Bytes In, Bytes Out, Protocol, HTTP Method, HTTP Version, URL Category, URL Hostname, URL Path, URL Query, Mime Type, File Name, User-Agent.

Below, I explained how we can use this information to hunt or detect threats.

Duration

This information shows how long the transaction has taken. Malware can communicate with the C2 server over the HTTP(S) protocol. When this is the case, it asks for commands periodically. This period doesn't have to be a constant value like every 10 minutes. Malware can also use jitter to make random-looking requests. Also, keeping the connection open can also be used by malware. In any case, it needs to either ask for commands very often or keep the connection open.

Technique

Calculate the sum per SourceIP-DestinationIP pair over 12/24 hours

What to look for

Higher values may indicate beaconing. Keep in mind that not all beacons are malicious. That's why we are hunting.
Note: If you apply the same method to your public websites, you can detect web scraping or customer data scraping.

HTTP Status

Users visit websites, post something, sometimes upload some data, or download a file. In normal conditions, these transactions have an HTTP 200 result. When it comes to malware, it is possible to use HTTP error codes as a C2 channel. Also, most malware use DGA(domain generation algorithm) in order to keep the connection persistent if one of the domains is blocked. In such a case, the malware keeps getting HTTP errors and tries the next domain.

Technique

1. Calculate the total count of the HTTP Status Codes per SourceIP or per SourceIP-DestinationIP over a specific time period.
2. List URLs having only HTTP Errors.

What to look for

1. Higher values of an uncommon HTTP Status Code may indicate C2 activity.
2. Higher values of HTTP errors for a website can indicate failed C2 activity.

Bytes In

In normal conditions, when a user visits a website, downloads a file, etc., each transaction has a different size. On the other hand, malware visits the same page every time. This makes the downloaded content has the same size unless the attacker starts interacting with the victim machine.

Technique

1. Calculate the count of BytesIn per Source-Destination pair over 12/24 hours. You have the best chance when the attackers sleep as there is no interaction.
2. Calculate the ratio of count(BytesIn) per Source-Destination pair. This is for comparing the attacker interaction versus idle status.

What to look for

1. Higher values may indicate beaconing. C2 servers reply with the same data, making Bytes In value the same.
2. Higher values of ratio may indicate C2 beaconing.

Bytes Out

A normal user activity consists mostly of downloading data. Uploaded data is usually small unless there is a file/data upload to a website.

Technique

1. Calculate the sum of BytesOut per Source-Destination pair over 12/24 hours.
2. Calculate the ratio of count(BytesOut) per Source-Destination pair over 12/24 hours.

What to look for

1. Higher values may indicate data exfiltration.
2. Higher values of ratio may indicate beaconing.

HTTP Method

In normal circumstances, a user's web traffic contains a large amount of HTTP GET, a small amount of HTTP POST methods. Other HTTP methods, such as HTTP PUT, are expected to be seen less.

Technique

Calculate the ratio of the POST or PUT over GET per Source-Destination over 4/8/12/24 hours.

What to look for

Higher values of ratio may indicate beaconing or exfiltration.

URL Hostname

Usually, a user visits websites that are in the top 1M list. In some cases, an unpopular website can be visited by lots of users as well (think about 3rd parties having business with the company).

Technique

1. Compare with top 1M domains and calculate the hit count.
2. Calculate hit count per Hostname.

What to look for

1. Hit count <5 and Hostname is not in the top 1M may indicate malicious payload delivery.
2. Small number of hit count may indicate malicious payload delivery.

URL Path

When an attacker compromises a website and uses it as a C2 server, the malware most probably uses the same URL Path for C2 communication.

Technique

Calculate count per Source-Destination-URLPath pair.

What to look for

Higher values may indicate beaconing.

URL Query

URL query information is seen when you search for an item on a website. Malware does the same when asking the C2 server if there is anything to run on the victim machine. The query can be encoded/encrypted as well.

Technique

1. Calculate count per Source-Destination-URLQuery.
2. Calculate the length of URLQuery.
3. Look for base64 encoded strings in URLQuery.

What to look for

1. Higher values may indicate beaconing.
2. Higher values may indicate encoded data, a sign of exfiltration or beaconing.
3. Encoded strings may indicate beaconing or exfiltration.

Mime(Content) Type

Unfortunately, most web proxies fail to determine the exact type of content.

Technique

List mime type per Source-Destination pair.

What to look for

Uncommon mime types may indicate a malicious file.

User Agent

Normally, all applications have their own user agent information. Malware can try to mimic a legitimate application user agent but sometimes fail to do that with a small typo.

Technique

Calculate count within the environment(long tail analysis).

What to look for

Lower values may indicate a malicious binary existence.

URL Category

In most environments, there are commonly blocked web categories like Hacking, Pornography, Dynamic DNS, etc. Uncategorized websites are a pain and sometimes this category has to be allowed for the sake of business continuity.

Technique

Query for Uncategorized, Dynamic DNS, and other suspicious categories. Calculate dcount of SourceAddress by URLHostname.

What to look for

Small dcount values may indicate abnormal/suspicious/malicious activity. If an uncategorized URL is visited by many users, it is less likely that the URL is malicious.

HTTP Version

There are four HTTP versions — HTTP/0.9, HTTP/1.0, HTTP/1.1, and HTTP/2.0. The current version is 1.1 and the future one will be 2.0.

Technique

Check HTTP versions

What to look for

HTTP/0.9 and HTTP/1.0 are old. This may be an indication of malicious activity.

Protocol

Web proxies are able to determine the protocol by analyzing the traffic.

Technique

Compare ports with their standard protocols.

What to look for

Common Protocol-Uncommon Port or Common Port-Uncommon Protocol may indicate malicious traffic.

File Name

It's not always possible to properly log the names of the files that are downloaded from the internet. If it's logged properly, file names can be used for hunting. Some malware droppers download randomly named files.

Technique

Entropy analysis on filenames.

What to look for

May indicate malicious payload delivery.

I hope you find this guide useful for your hunts! I've also created a cheat sheet and shared it in my GitHub repo. You can find the pdf version here.

Thanks for reading this article! If you have any questions, leave a comment below. Want to master KQL for Threat Hunting, Detection Engineering, and DFIR in a hyper-realistic environment? Visit my academy for a free course!

Mehmet is the founder of Blu Raven Academy. He brings over 15 years of experience in cybersecurity, with a unique blend of expertise in KQL, threat hunting, detection engineering, and data science to his courses to help others advance their skills. Recognized four times as a Microsoft Security MVP, he is renowned for adapting the RITA beacon analyzer to KQL, developing novel methods for detecting threats, and for his insightful presentations at key conferences like the SANS DFIR Summit.