I barely remember a security incident, that I worked on, where the adversary didn’t use web shells here or there. Web shells are effective, publicly available, and sort of hard to discover — they say.
In this piece, I will try to address the discovery challenge by sharing ideas and techniques for web shells hunting and ultimately turn the web shell from a capability to a liability for the adversary.
Introduction
Web shell is any form of code (whether scripted or compiled) that provides the adversary access, over web, to the operating system services (e.g. shells). Web shells usually provide services like command execution, file read/write, tunneling, just to mention few.
ASPXSpy, ReGeorg, Antak, and China Chopper are samples of a long list of publicly available web shells with varying capabilities.
Web shells are used by adversaries in different stages of the intrusion life cycle for different purposes.
- Establish Foothold: following the initial compromise, the adversary needs to create an opening through which (s)he can control the compromised system and upload additional tools. Web shell is a go-to option when the initially compromised system is an Internet-facing web server.
- Escalate Privilege: web shells inherent the privilege of the user that’s running the web service. If the web service runs with a privileged account whether misconfigured or by default off-the-shelf, it will give the adversary privileged access on the system. For example, Microsoft Exchange IIS worker processes run as “NT AUTHORITY\SYSTEM”.
- Maintain Persistence: adversaries use web shell on Internet-facing servers to maintain access to the environment. Web shells are passive, don’t phone up home but rather wait for adversary’s knocks which makes them stealthy — when aren’t in use — and good option for persistence. Moreover, an adversary with a left over web shell may survive enterprise-wide password reset because the web shell often runs in the context of web service’s password-less, system-managed service account.
- Move Laterally: web shells, with networking capabilities like tunneling and port forwarding, are used by adversaries to pivot and move to other systems in the target network.
- Complete Mission: web shells with file download or SQL statement execution capabilities are used by adversaries to exfiltrate data.
As I mentioned, web shells come in two forms:
— Web script: ASP.NET/PHP/JSP
— Compiled code: whether native or managed code
This piece mainly addresses the script-based web shells. However, majority of the hunting techniques, that going to be discussed, apply to the compiled web shells as well. A separate blog post will come for compiled web shells, stay tuned.
Before we start, keep in mind that when I say hunting I mean the real discovery which doesn’t rely on any the prior knowledge of the web shell itself (the code or signature). It’s not the signature-based detection of the web shell which requires a prior knowledge of the web shell that the adversary fully controls and thus can manage to evade detection. In this piece we will depend on sources that less controlled by the adversary and that provide a broader coverage regardless of whatever specific technique the adversary used in the web shell.
In games, you should not rely on what the adversary controls but you. Otherwise, be prepared for surprises.
Now let’s go hunt for web shells
First of all, let me highlight the need of initially identifying the systems that run web service in your environment — especially the ones that are exposed to the Internet or any extranet — and the web application directories. That’s the hunting scene.
The data sources that we will rely on for hunting are:
- File system metadata and audits
- Process creation audits
- Web service access logs
I will leave the data collection and processing to you and jump directly into the hunting.
File system metadata and audits
Web shells, just like any other file on the system, do have records associated with them in the file system metadata. These records contain information; like filename, timestamps, ownership, and inode; that is of great value for web shell hunting.
In a typical web application, you’ll see clusters of files and directories created/modified around the same time period by the same service or application account and the inodes are almost sequential. This is what you would expect to see in a typical web application server, right!. The web service installed, then the web application installed, and every now and then web application was updated, these are the clusters.
In above snippet (Figure 3), you can easily tell: when the IIS was installed, the user installed it, and the sequence of the files’ inodes. With such context, it will be very hard for an arbitrarily created file, a web shell, not to show up as an outlier with an inconsistent timestamps, owner, or inode.
In the following snapshot (Figure 4), you can easily spot MSExchange.aspx as an outlier which should be investigated. The file’s timestamps, owner, and inode don’t blend in well with the rest in the directory. Note: just for the ease of illustration, I filtered out the data set to include only OWA’s ASP.NET-related files, however the more data you have, the easier you can spot outliers.
Also, whenever you identify a web shell, look for other files created/modified around the times of the web shell or created by the same user that created the web shell. That will likely reveal more (e.g. malware, tools, output of the commands executed by the adversary, …)
But, wait! What if the adversary decided not to create a standalone web shell but inject the code into one of the already existing legitimate files? That’s what I used to see adversary doing to avoid discovery and, indeed, it would defeat the aforementioned technique. In such cases, you should look for outliers in $STANDARD_INFO Modified and Changed timestamps — the ones that tick when the file content is modified on Windows systems for example. However, always remember that $STANDARD_INFO timestamps are not as reliable as the $FILE_NAME and they can be easily tampered using Win32 API.
What then? Defeat? Nah! in such cases, you just need to audit modifications being made on the files in the web application directories and hunt for unauthorized modifications. File Integrity Monitoring (FIM) solutions come to play here. But if FIM is not feasible, you can enable file system auditing — perhaps targeting only successful write events to avoid performance issues — on the web application directories and then look into Security events (EID 4656) for handle requests with WriteData access to an already existing File objects (see Figure 5). The contextual information that 4656 events provide — the user and the process that made the change — plus your knowledge about your web application would help you triage the events and identify the suspicious modifications which should be investigated.
Process creation audits
One of the common services, that web shells provide, is command execution. When a command is executed through a web shell, the command’s process will be created by the web service worker process (e.g. w3wp, http.exe), and this is what we will search for.
In Windows, process creation audits (Security EID 4688 and Sysmon EID 1) are the data sources we will use for this hunt. Native process auditing (EID 4688) is not enabled by default. Here is Microsoft’s guide on how to enable process creation auditing and include the process’s command-line which won’t be logged by default. The process’s command-line auditing requires (KB3004375) to be installed for Windows versions prior to Windows Server 2012 R2.
Figure 6 lists the event 4688 versions, the Windows Server versions support it, and the status of creator process’s information required for the hunt.
4688 version 0 (Figure 7) doesn’t log the parent ID nor the name, thus the hunting technique doesn’t apply to this version. You need to upgrade or easier perhaps install Sysmon.
4688 version 1 (Figure 8) does log the ID of the creator process but not the name. Thus you need to correlate with the preceding events to identify the name of the creator process.
4688 version 2 (Figure 9) does log all the field we need for the hunt.
Sysmon EID 1 (Figure 10) does log what we need for this hunt and much more. Whether for detection, response, or hunting purposes; I can’t emphasize enough on having Sysmon installed, at least, on your critical systems.
Now using EID 4688 v2 or Sysmon EID 1, the stacking of the processes created by the web service worker processes can tip you off the presence of a web shell that’s capable of executing commands. The frequency and the type of the created processes are the features that you will use to analyze the stacked data.
The following PowerShell script reads all Sysmon EID 1 events where the creator process name is the web service worker process (w3wp.exe in this case), and then stacks the child processes based on the process Image.
$Events = Get-WinEvent -FilterHashtable @{Logname='Microsoft-Windows-Sysmon/Operational';Id=1}
$Out = ForEach ($Event in $Events) {
$eventXML = [xml]$Event.ToXml()
# If the parent's Image is the IIS worker w3wp.exe
if ($eventXML.Event.EventData.Data[19].'#text' -Like '*\w3wp.exe'){
# Print the process Image
$eventXML.Event.EventData.Data[4].'#text'.ToLower()
}
}$Out | Group-Object -NoElement | Sort-Object count | Format-Table count, name
The following snippet (Figure 11) shows the script output from the IIS server I set up for demo. The highlighted commands were the commands I executed on the system via a web shell. The nature of the commands, the frequency of execution, and the fact that they were created by the web service, are quite suspicious!
Now let investigate these processes further by looking at the full command line. Let’s slightly modify the script to list the CommandLine.
$Events = Get-WinEvent -FilterHashtable @{Logname='Microsoft-Windows-Sysmon/Operational';Id=1}
$Out = ForEach ($Event in $Events) {
$eventXML = [xml]$Event.ToXml()
# If the parent's Image is the IIS worker w3wp.exe
if ($eventXML.Event.EventData.Data[19].'#text' -Like '*\w3wp.exe'){
# Print the process CommandLine
$eventXML.Event.EventData.Data[9].'#text'.ToLower()
}
}$Out
Below is the output, the list of the full command line of the processes executed by the IIS workers.
Highlighted commands should set off alarm bells, shouldn’t they?
However, the presence of a web shell, with command execution capability, is not the only hypothesis for such a symptom, web service spawning suspicious processes. Direct exploitation of the web service/application vulnerability is another hypothesis that you should consider if there’s no indicators of web shell presence — or any other backdoor.
Web service access log
Web service access log records the requests handled by the web service. The records include the requests’ timestamp, URI, client IP, referrer, User Agent (UA), HTTP status code, among other things. Stacking of such information, searching for anomalies and the least frequent entries can reveal the presence of web shell and even other suspicious activities (scans, exploitation, …). Access logs can be analyzed using text-processing tools like grep/awk/uniq/sort but if you looked for a better tool to parse access logs, I would recommend Microsoft’s Log Parser, and its Studio, the graphical user interface.
Now let’s start…
URI
This hunt is based on the fact that web shells are not usually requested as frequent as the legitimate resources served by the web server. This hypothesis in addition to your knowledge about your web application are good start for the hunt. Keep in mind though, you may stumble upon stuff that is rarely requested but still legitimate — requested on a rare legitimate conditions.
With that in mind, stack the URI’s recorded in the access logs and then look at the least frequent requests and review the content and the metadata of the associated files.
The following is the Log Parser SQL statement to stack the URI field from the IIS logs.
SELECT cs-uri-stem AS URI, COUNT(*) AS Hits
FROM '[LOGFILEPATH]'
GROUP BY cs-uri-stem ORDER BY HitsFigure 13 shows the result, the stacked URIs from my testing environment. The highlighted entry is the web shell. Note: the IIS logs were from a testing environment where the web shell was used way more frequently than the actual web application 🙃.
Also look at the requests to resources with weird filename, either the name (e.g. 1–3 chars) or the extension that is not expected (e.g. .txt). Review the content and the metadata of the files associated with these requests.
Here, the Log Parser SQL statement that stacks the URI field for the requests to resources that their file name is 1–3 characters.
SELECT cs-uri-stem As URI, COUNT(*) As Hits
FROM '[LOGFILEPATH]'
WHERE cs-uri-stem LIKE '%/_.___' OR cs-uri-stem LIKE '%/__.___' OR cs-uri-stem LIKE '%/___.___' OR cs-uri-stem LIKE '%/_' OR cs-uri-stem LIKE '%/__' OR cs-uri-stem LIKE '%/___'
GROUP BY cs-uri-stem
ORDER BY HitsAnd this is the Log Parser SQL statement that stacks the extensions.
SELECT EXTRACT_EXTENSION(cs-uri-stem) As Extension, COUNT(*) As Hits
FROM '[LOGFILEPATH]'
GROUP BY Extension
ORDER BY HitsReferer
HTTP referer is the resource that the user last visited, the one that provided a link to the current resource. Mostly, an implanted web shell would be accessed directly by the adversary and won’t be linked in a legitimate page. With that in mind, stack the URI’s of the requests that have no referer field. Triage the files associated with the least frequent requests and don’t have the referer set.
SELECT cs-uri-stem AS URI, COUNT(*) AS Hits
FROM '[LOGFILEPATH]'
WHERE cs(Referer) IS NULL
GROUP BY cs-uri-stem ORDER BY HitsClient IP
For those who expect users from specific countries, collect the unique client IP addresses and look for requests from unexpected countries, and review the resources they accessed.
SELECT DISTINCT c-ip FROM '[LOGFILEPATH]'If the web server is behind a reverse proxy, you will need to configure X-Forwarded-For on the reverse proxy and the web server to get the client real IP address.
Hope you enjoyed the blog post. Happy hunting! 👋🏾
UPDATES:
2019–06-28: Josh Bryant did a nice work on automating some of the filesystem hunt. He also gave two interesting talks on the topic. 👍🏾
