Detection of web vulnerability scanning with PHP and Apache mod_status.

Photo by Markus Spiske on Unsplash

Hello! It’s been a while, but we’re back. Today we will talk about a topic associated with mitigating attacks on your websites. If you are a server administrator or manage your own server to host your website, I hope it is helpful.

During the last few months I have been noticing some load spikes on our servers, when investigating the cause I found a pattern in them, the infamous and widely used web vulnerability scanners.

If you are not familiar with the concept, these types of tools have dictionaries of URLs or names of scripts with known vulnerabilities or files that a careless administrator can leave visible from the Internet to an attacker and, when found, are reported for future attacks to through already known vectors.

Now, you can have a WAF and Firewall that are very helpful but are not very effective for simple requests in which you try to discover whether or not a URL or script exists on a website (for malicious purposes) and there This is where something additional is required.

The solution

Since I did not have a mechanism to stop this type of behavior in which large amounts of requests are made to multiple websites from different IPs trying to “guess” vulnerable URLs, I began to consider some type of script that would help me monitor, block and leave a record of this type of activity, mitigating possible load peaks on my beloved servers.

Functional requirements:

• Query the requests made to the web server on a constant basis.
• Analyze each URL in the requests from rules associated with the content (text) in the URL or number of requests made by the same IP.
• Perform one or more actions from the information found, in this case block the IP in our firewall

Let’s explore the technical requirements below.

Technical Requirements

• Apache HTTP Server
• mod_status enabled on Apache HTTP Server
• PHP 8.3 cli. Yes, we are going to use PHP in console mode for this guide.
• Firewall (optional), in this case we will use ufw on Linux Mint 21.2

Apache HTTP Server

Check if you have Apache installed on your PC or on the environment where you are going to perform your tests. In my case I am using Linux Mint 21.2, if you don’t have it installed, you can do it using the following command (in my version of Linux, check your OS version to run the correct command):

sudo apt install apache2

You can check the installation guide for Ubuntu based versions at the following link:

How to install Apache2 | Ubuntu

Once installed verify that it is running, you can access with your browser http://localhost/ and you should be shown a welcome message from Apache HTTP Server.

Apache localhost

If it is not running don’t forget to start the service. Generally you can do this by running sudo systemctl start apache2

mod_status

The second step to move forward is to have a source of information where I can constantly query the requests made to my web server, and that’s where mod_status comes in handy.

mod_status is an Apache HTTP Server module that allows an administrator to query at any time its status and the requests being served at that instant.

Information about mod_status can be found at the following link:

mod_status - Apache HTTP Server Version 2.4

Let’s check if the module is active in our web server, this can be done by executing the following command:

apachectl -M

Once executed, the list of modules loaded in the web server will be shown, there you can confirm if “status_module (shared)” is found, if so, it is active. You can also confirm it by going to http://localhost/server-status

You should see something similar to this:

In case you cannot find the module, you can load it using the a2enmod command like this:

sudo a2enmod status

IMPORTANT: If you are going to enable mod_status in a production environment, make sure that it is only visible from authorized IPs and domains, normally access is allowed only to requests coming from localhost, but it is something I advice to keep in mind.

PHP 8.3 cli

Verify that you have PHP installed on your PC, the source code of the scripts is compatible with PHP 8.x. If you have a previous version you may not require changes and it will work as expected.

You can check if you have PHP in cli mode by running the command below:

php -v

Firewall

The firewall on your PC is optional, but the general idea of the script is to run a command to send to the firewall the source IP of the request and block it. In this case we will use ufw, you can learn more about this firewall for Ubuntu at the following link:

Firewalls | Ubuntu

And these are the requirements, let’s get to the solution!

Photo by Avi Richards on Unsplash

Create a directory to store the files we are going to use, the structure will be as follows:

Files

Now let’s look at the function of each file, let’s start with our detection engine.

Scanengine.php

This file contains two functions, the first one “getApacheStatus()” allows to query and process the mod_status response using cURL and the HTML processing library DomDocument. Here we do some scraping, we look for a tag that identifies the table in which the requests are found and extract the records in plain text into an array.

IMPORTANT: do not forget to activate these modules in case your PHP installation does not have them by default.

The second function is “checkRequests()” and is responsible for taking the requests found and processing them using the rules and actions predefined in the constructor of the class “$scanRules” and “$scanActions”.

Here is the source code:

<?php

/**
 * Scanengine Class
 * 
 * A class for scanning the state of the Apache server and applying rules.
 */
class Scanengine
{
    /** @var array $currApacheRequests An array to store the current requests to the Apache server. */
    private array $currApacheRequests;

    /** @var string $apacheURL The URL of the Apache server status page. */
    private string $apacheURL;

    /** @var array $scanRules An array containing the scanning rules. */
    private array $scanRules;

    /** @var array $scanActions An array containing actions to take based on scan results. */
    private array $scanActions;

    /**
     * Scanengine constructor.
     * Initializes class properties and sets default values.
     */
    public function __construct()
    {
        $this->currApacheRequests = array();
        $this->apacheURL = "http://localhost/server-status";

        $this->scanRules = array(
            "scripts" => function (array $currApacheRequests): void {
                echo PHP_EOL . "-- Analysis using text string dictionary --" . PHP_EOL;

                // Load the list of terms and words to search in URLs for scanning suspicious requests
                $scriptDictionary = file("scripts.txt", FILE_SKIP_EMPTY_LINES | FILE_IGNORE_NEW_LINES);

                if (!is_array($scriptDictionary)) {
                    echo "There was an error reading scripts.txt, please verify if the file exists in the root folder." . PHP_EOL;
                    return;
                }

                $hits = 0;

                foreach ($scriptDictionary as $id => $script) {
                    foreach ($currApacheRequests as $reqId => $request) {
                        if (str_contains($request["Request"], $script)) {
                            echo PHP_EOL . "Match found for suspicious script '{$script}' in request '{$request["Request"]}' from IP {$request["Client"]}" . PHP_EOL;

                            $this->scanActions["firewallBlock"]($request["Client"]);
                            $hits++;
                        }
                    }
                }

                echo PHP_EOL . "Found {$hits} hit(s) in the analyzed requests." . PHP_EOL;
            },
            "IPHits" => function (array $currApacheRequests): void {

                // IP hits scanning
                $maxHits = 3;
                $IPHits = array();

                echo PHP_EOL . "-- Analysis of hit count per IP, Maximum {$maxHits} hit(s) --" . PHP_EOL;

                foreach ($currApacheRequests as $reqId => $request) {
                    if (!isset($IPHits[$request["Client"]])) {
                        $IPHits[$request["Client"]] = 1;
                    } else {
                        $IPHits[$request["Client"]]++;
                    }
                }

                $hits = 0;

                foreach ($IPHits as $ip => $hitsCount) {
                    if ($hitsCount > $maxHits) {
                        echo PHP_EOL . "Allowed hit count exceeded ({$maxHits}) from IP {$ip}" . PHP_EOL;
                        $this->scanActions["firewallBlock"]($ip);
                        $hits++;
                    }
                }

                echo PHP_EOL . "Found {$hits} hit(s) in the analyzed requests." . PHP_EOL;
            }
        );

        $this->scanActions = array(
            "firewallBlock" => function (string $clientIP): void {
                // Don't shoot yourself in the foot.
                $arrIgnoreIps = array("127.0.0.1", "::1");

                if (!in_array($clientIP, $arrIgnoreIps)) {
                    echo "IP {$clientIP} added to firewall blacklist!" . PHP_EOL;
                }
            }
        );
    }

    /**
     * Sets the URL of the Apache server status page.
     * 
     * @param string $apacheURL The URL of the Apache server status page.
     */
    public function setApacheURL(string $apacheURL): void
    {
        $this->apacheURL = $apacheURL;
    }

    /**
     * Retrieves the state of the Apache server and analyzes requests.
     * 
     * @return void
     */
    public function getApacheStatus(): void
    {
        $url = $this->apacheURL;

        $curl = curl_init($url);
        curl_setopt($curl, CURLOPT_URL, $url);
        curl_setopt($curl, CURLOPT_HEADER, false);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

        $response = curl_exec($curl);

        if ($response === false) {
            echo 'Curl Error: ' . curl_error($curl) . PHP_EOL;
            exit;
        }

        curl_close($curl);

        $dom = new DomDocument();
        @$dom->loadHTML($response);

        $tables = $dom->getElementsByTagName('table');
        $foundRequests = false;

        foreach ($tables as $table) {
            foreach ($table->attributes as $attr) {
                if ($attr->name == "border" && $attr->value == 0) {

                    echo "List of requests found at {$this->apacheURL}" . PHP_EOL;

                    $foundRequests = true;

                    $fieldHeaders = $table->getElementsByTagName('th');
                    $arrFieldKeys = array();

                    foreach ($fieldHeaders as $fieldHeader) {
                        $arrFieldKeys[] = $fieldHeader->nodeValue;
                    }

                    $requests = $table->getElementsByTagName('tr');

                    foreach ($requests as $req) {
                        $arrFields = array();
                        $fields = $req->getElementsByTagName('td');

                        if (count($fields) == 0) {
                            continue;
                        }

                        foreach ($arrFieldKeys as $key => $tag) {
                            $arrFields[$tag] = str_replace("\n", "", $fields->item($key)->nodeValue);
                        }

                        if (!empty($arrFields)) {
                            $this->currApacheRequests[] = $arrFields;
                        }
                    }
                }
            }

            if ($foundRequests) {
                break;
            }
        }

        if (!$foundRequests) {
            echo "List of requests at {$this->apacheURL} not found, perhaps an incorrect URL?" . PHP_EOL;
        }
    }

    /**
     * Executes rule-based checks on Apache server requests.
     * 
     * @return void
     */
    public function checkRequests(): void
    {
        $this->scanRules["scripts"]($this->currApacheRequests);
        $this->scanRules["IPHits"]($this->currApacheRequests);
    }
}

As you can see in the scan rules array “$scanRules”, the purpose of this array is to store the routines that analyze the detected requests in the URL /server-status, you can create as many as you need. For this article we implemented the detection rule using a dictionary of script/text names within a URL and number of simultaneous hits per source IP, which can also indicate a malicious scan to your server.

In the “$scanActions” array we implement an action that simulates the execution of a block on the server’s firewall based on what is found by our scanning rules.

IMPORTANT: To perform this type of actions, such as adding an IP in the server blacklist, your php script generally must be executed as root or with a user that has permissions for this operation, remember that executing scripts or processes with root permissions can be risky.

If you want to add a call to the firewall to block the IP you can do it by including a line like this in the body of the “firewallBlock” function in “$scanActions”:

$this->scanActions = array(
    "firewallBlock" => function (string $clientIP): void {
        $result = shell_exec("ufw deny from {$clientIP} to any");
        echo $result.PHP_EOL;
        echo "IP {$clientIP} added to the firewall blacklist!" . PHP_EOL;
    }
);

This is the heart of our solution, now that we have analyzed it let’s move on to the two remaining files, scripts.txt and apache-scan.php

scripts.txt

This file stores the list of text strings used in dictionary attacks to our web sites, for this case we will use some terms as shown below:

admin.php
Exchange.asmx
phpMyAdmin
backup
phpmyadmin3
xx.php
backup.zip
dump.sql
config.php
.env.production
web.config
config.xml
config.yml
sql.sql.xz
backups.sql.zip
web.sql.zip
website.sql.zip
backup_4.sql.zip
fm1.php?x=zourt
www.sqlitedb
.git/HEADE

These strings are loaded at the beginning of the execution of the “scripts” rule and each term is searched in the request list for matches.

IMPORTANT: for a large number of terms in a dictionary, a more efficient search strategy than the one presented here should be considered.

apache-scan.php

Finally, let’s analyze this file, which is the entry point of our solution and is responsible for creating an instance of Scanengine and calling the functions of our detection engine.

<?php
require("Scanengine.php");

$scan = new Scanengine();
$scanInterval = 5;

do {

    $scan->getApacheStatus();
    $scan->checkRequests();
    sleep($scanInterval);
} while (true);

As you can see, it also has a cycle and waits for a certain number of seconds, so that the request scan can be performed continuously.

IMPORTANT: this solution does not have an option to check if you have already applied a block to an IP and there may be scenarios where you try to perform a block already executed if the request to Apache is still visible in mod_status.

Now with our solution ready, we just need to run apache-scan.php like this:

php <script route>/apache-scan.php

If all goes well, you should see messages like these:

List of requests found at http://localhost/server-status

-- Analysis using text string dictionary --

Match found for suspicious script 'gifLogo1.gif' in request 'GET /classic/img/gifLogo1.gif HTTP/1.1' from IP x.x.x.x
IP x.x.x.x added to the firewall blacklist!

Found 1 hit(s) in the analyzed requests.

-- Analysis of hit count per IP, Maximum 3 hit(s) --

Allowed hit count exceeded (3) from IP x.x.x.x
IP x.x.x.x added to the firewall blacklist!

Allowed hit count exceeded (3) from IP y.y.y.y
IP y.y.y.y added to the firewall blacklist!

Found 2 hit(s) in the analyzed requests.

And that’s all, remember that in Winkhosting.com we are much more than hosting!