Nmap Detection with Wireshark

Introduction

The following article’s objective is to teach the reader how to detect and analyze all the different Nmap scan types by using Wireshark. First, let’s introduce both tools.

Wireshark

Wireshark is a free and open-source packet analyzer. It is commonly used for network troubleshooting, analysis, software and communications protocol development, and education. In the cybersecurity sector, Wireshark is used to monitor and analyze traffic files in order to identify anomalies in the network behavior.

Wireshark logo

Nmap

Nmap (“Network Mapper”) is a free and open source utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics.

It was designed to rapidly scan large networks, but works fine against single hosts. Nmap runs on all major computer operating systems, and official binary packages are available for Linux, Windows, and Mac OS X.

Nmap Logo

Unfortunately, Nmap is widely used in the cyberattacker communities as one of the main reconnaissance(TA0043) tools, specially in terms of port scanning. Nmap allows the attackers to carry out mass port scanning activities.

There exist different type of scans that can be made with Nmap. As stated, the purpose of this article is to explain one by one all of them as well as demonstrating how can they be detected thanks to an easy analysis with Wireshark. This article will also provide the readers with the used Wireshark’s filters so they can use them in their own environments.

Relevant information to understand the article

The below practical is performed by the simulation of two endpoints. Endpoint A is the one that will act as the Attacker (Nmap). While, Endpoint B is the one that will represent the victim (Wireshark):

• Endpoint A: Its operative system is Kali Linux and its IP address for this experiment is 192.168.1.5
• Endpoint b: Its operative system is Ubuntu Linux and its IP address for this experiment is 192.168.1.4

Normally, Nmap scans are carried out from an external IP since the attacker does not have internal access to the network. Given the case where this situation is detected internally, that would mean that the attacker has already achieved to set a foothold inside the network.

TCP Scan

This type of Scan uses the three-way handshake of the TCP to discover if a port is open. Basically, the attacker will begin a TCP connection by sending a package with the (SYN) flag on, as well as indicating to which destination port it wants to connect. From here, there will be two options:

• Open port: If this port is open in the client, this will answer with a package, where its (SYN,ACK) flags are on. Finally, the computer’s computer will accept the connection (ACK) to then reset it (RST, ACK).

Diagram 1: TCP scan results in an open port

• Closed port: If the port is closed, the client will answer the (SYN) request with an answer where the (RST, ACK) flags will be on.

Diagram 2: TCP scan results in closed port

To carry out this type of scans, we need to type the following nmap command:

nmap -sT -p PORT_NUMBER DEST_IP_ADDRESS

Example

I carried out this example in a Kali based system that was scanning an Ubuntu system that hosted Wireshark. I opened port number 22 (ssh), while I had port number 400 closed. The results were the following:

Nmap scan that results in an open port (22)

When the port is open, we can easily see that the attack IP address generated the [RST,ACK]. So, anytime we get a completed three-way handshake that ends with a TCP packet which set flags are [RST,ACK], we will be able to relate it to a NMAP TCP scan.

On the other hand, when the port is closed, we get the following Wireshark output:

Nmap scan that results in a closed port (4000)

Anytime we get only the start of a three-way handshake (SYN flag) followed by TCP packet which set flags are [RST,ACK], we will be able to relate it to a closed port.

The used Wireshark filter to detect the existence of a nmap TCP scan in both cases has been:

tcp and tcp.flags == 0x014

TCP flags equal to 0x014 equals to (ACK,RST) flags set. The justification for this filter is that every time a TCP scan is carried out, even if it is successful or unsuccessful, one of the two endpoints involved returns a packet with those two flags enabled. It is the duty of the analyst to check if the nmap scan was successful or not. If it was, the mentioned package will be returned by the attacker IP address. On the other hand, the mentioned package will be returned by the victim IP address.

Stealth scan

A Stealth Scan in Nmap, often referred to as a SYN scan or half-open scan, is a type of port scanning method that aims to minimize the chances of being logged while still providing essential information about open ports on a target system. It is considered stealthy because it does not complete the full three-way TCP handshake.

SYN scan is the default and most popular scan option for good reasons. It can be performed quickly, scanning thousands of ports per second on a fast network not hampered by restrictive firewalls. It is also relatively typical and stealthy, since it never completes TCP connections. Furthermore, it can have two possible results:

• Open port: The victim’s endpoint will return a TCP packet which set flags will be [SYN,ACK], since the endpoint intention is to complete the three-way handshake.

Diagram 3: Stealth scan results in an open port

• Closed port: The victim’s endpoint will return a TCP packet which set flags will be [RST,ACK].

Diagram 4: Stealth scan results in closed port

To carry out this type of scans, we need to type the following nmap command:

nmap -sS -p PORT_NUMBER DEST_IP_ADDRESS

Example

I carried out this example with the same settings as the past example. The results were the following:

When the port is open, we can easily see that the attack IP address generated the [RST]. So, anytime we get an unsuccessful three-way handshake that ends with a TCP packet which set flag is [RST] from the IP address that started it, we would be able to relate it to a stealth nmap scan.

On the other hand, when the port is closed, we get the following Wireshark output:

Anytime we get only the start of a three-way handshake (SYN flag) followed by TCP packet which set flags are [RST,ACK], we will be able to relate it to a closed port.

The used Wireshark filter to detect the existence of a Nmap Stealth Scan — Open Port is:

tcp and tcp.flags == 0x004

TCP flags equal to 0x004 equals to [RST] flags set. The justification for this filter is that every time a Stealth scan is carried out, and it finds an open port, the attacker’s computer will return a TCP packet with RST flag enabled. The analyst will have to check that this package is sent as a response of the victim’s response to the start of a three-way handshake, [SYN, ACK] package.

However, if the port is closed, we can use the same filter as we did in last scan type:

tcp and tcp.flags == 0x014

In this case, the analyst will have to check that this package is sent as a response of the beginning of a three-way handshake, [SYN] package.

FIN scan

A FIN scan is a type of port scanning method in Nmap that uses the TCP FIN flag to determine the state of a target port. The FIN (Finish) flag is normally used to gracefully close an established TCP connection.

Observation: FIN-Scans are only workable in Linux machines and does not work on the latest version of Windows.

This type of scan can have two results:

• Open port: The victim’s endpoint will not return any response.

Diagram 5: FIN scan results in open port

• Closed port: The victim’s endpoint will return a response in which TCP [RST, ACK] flags will be set.

Diagram 6: FIN scan results in closed port

To carry out this type of scans, we need to type the following nmap command:

nmap -sF -p PORT_NUMBER DEST_IP_ADDRESS

Example

I carried out this example with the same settings as the other examples. The results were the following:

FIN scan results in an open port

When the port is open, we can easily see that the attack IP address generated the [FIN]. So if this requests receives no response, that would mean that the port is open.

On the other hand, when the port is closed, we get the following Wireshark output:

FIN scan results in a closed port

Anytime we get a response with [RST, ACK] flags set, that would mean that the port is closed.

The used Wireshark filter to detect the existence of a Nmap FIN Scan is:

tcp and tcp.flags.fin

This filter will show all the Wireshark packets that contain the [FIN] flag set or simply respond to a packet that had this flag set. The analyst will have to check that the detected packages are responded (closed port) or not (open port).

NULL Scan

A Null scan is a type of port scanning method in Nmap that involves sending TCP packets with no flags set (hence, a “null” scan).

Observation: NULL-Scans are only workable in Linux machines and does not work on the latest version of Windows.

This type of scan can have two results:

• Open port: The victim’s endpoint will not return any response.

Diagram 7: NULL Scan results in open port

• Closed port: The victim’s endpoint will return a response in which TCP [RST, ACK] flags will be set.

Diagram 8: NULL Scan results in closed port.

To carry out this type of scans, we need to type the following nmap command:

nmap -sN -p PORT_NUMBER DEST_IP_ADDRESS

Example

I carried out this example with the same settings as the other examples. The results were the following:

NULL scan results in an open port

When the port is open, we can easily see that the attack IP address generated a TCP packet with no flags set [<None>]. So if this requests receives no response, that would mean that the port is open.

Contrary to this, when the port is closed, we get the following Wireshark output:

NULL scan results in a closed port

Anytime we get a response with [RST, ACK] flags set to a request with no flags set [<None>], we will know that a Null Scan has been carried and has returned a closed port value.

The used Wireshark filter to detect the existence of a Nmap NULL Scan is:

tcp and tcp.flags == 0x000

This filter will show all the Wireshark TCP packets that have no flags set. The analyst will have to check that the detected packages are responded (closed port) or not (open port).

XMAS Scan

These scans are designed to manipulate the PSH, URG and FIN flags of the TCP header, Sets the FIN, PSH, and URG flags, lighting the packet up like a Christmas tree.

XMAS set flags

When source sent FIN, PUSH, and URG packet to a specific port and if the port is open then destination will discard the packets and will not send any reply to the source.

Observation: NULL-Scans are only workable in Linux machines and does not work on the latest version of Windows.

This type of scan can have two results:

• Open port: The victim’s endpoint will not return any response.

Diagram 9: XMAS scan results in an open port

• Closed port: The victim’s endpoint will return a response in which TCP [RST, ACK] flags will be set.

Diagram 10: XMAS scan results in an closed port

To carry out this type of scans, we need to type the following nmap command:

nmap -sN -p PORT_NUMBER DEST_IP_ADDRESS

Example

I carried out this example with the same settings as the other examples. The results were the following:

NULL scan results in an open port

When the port is open, we can easily see that the attack IP address generated a TCP packet with no flags set [<None>]. So if this requests receives no response, that would mean that the port is open.

Contrary to this, when the port is closed, we get the following Wireshark output:

NULL scan results in a closed port

Anytime we get a response with [RST, ACK] flags set to a request with no flags set [<None>], we will know that a Null Scan has been carried and has returned a closed port value.

The used Wireshark filter to detect the existence of a Nmap NULL Scan is:

tcp and tcp.flags == 0x000

This filter will show all the Wireshark TCP packets that have no flags set. The analyst will have to check that the detected packages are responded (closed port) or not (open port).

UDP SCAN

UDP scans in Nmap are used to discover open UDP (User Datagram Protocol) ports on a target system. UDP is a connectionless protocol, and services that use UDP may be harder to discover than those using TCP. Nmap provides several UDP scanning techniques to probe for open UDP ports. As all the other types of scans, the UDP scan can only have 2 types of results:

• Open Port: The victim’s endpoint will return a response with port data.

Diagram 11: Open UDP port

• Closed port: The victim’s endpoint will return an ICMP response, stating that the ICMP Port is unreachable.

Diagram 12: Closed UDP port

To carry out this type of scans, we need to type the following nmap command:

nmap -sU -p U:X,Y,Z target_ip

Example

I carried out this example with the same settings as the other examples. The results were the following:

UDP scan results in an opened port

Contrary to this, when the port is closed, we get the following Wireshark output:

UDP scan results in a closed port

Anytime we get an ICMP response with Destination unreachable set, preceeded by an UDP request, we will know that an UDP Scan has been carried and has returned a closed port value.

The used Wireshark filter to detect the existence of a Nmap UDP Scan is:

icmp.code == 3

This filter will show all the Wireshark ICMP packets that result in Port unreachable. It will be out job to later filter them to see if they answer an UDP request.