Install a Real-Time Intrusion Detection System (IDS) with Suricata and Python

Meerkats (suricatos)

This period was supposed to be a sabbatical, but for the last 2 weeks I’ve been attending the Google Cybersecurity Certificate course on Coursera, and I’ve been learning about foundations of Cybersecurity, how to address vulnerabilities and manage risks, how networks are composed, traffic flows, how to secure networks, protect assets and manage alerts with SIEM (Security Information and Event Management).

I had previous red team experience with pentesting Active Directory and Web apps with TCM Security (it was awesome). This certificate showed me the other side of the coin, the blue team, a more organized approach on how to defend infrastructure following Playbooks, legislation and documenting everything.

https://www.coursera.org/professional-certificates/google-cybersecurity?productTypeDescription=Professional%20Certificates&=null

One course that I found to be very interesting was the Networks and Network Security. At first, I had little knowledge about it. But I knew it was a very important topic.

One of the cornerstones of cybersecurity is the layered or multi level security, which includes reduction of attack surface, to minimize risks of a system compromise. One of the ways to reduce this attack surface is to use firewalls: between the modem and the router, given that the router will distribute internet signal from the modem to our devices, and also firewalls in the Operating System. Other important concepts are the separation of duties (a data engineer will access to databases, but not to service accounts of Machine Learning pipelines), role-based access control (segmentation of access scopes, like finance and software development) and the principle of least privilege (for instance, you won’t allow admin access to an user, unless it is strictly necessary).

reduction of attack surface

separation of duties

role-based access control

principle of least privilege

In order to increase security of our personal computer we can apply firewalls to block traffic on unused ports (ufw command in Ubuntu), create a strong password for login, apply authorization restrictions in some folders (chmod command in Ubuntu), keep the computer up-to-date with patches, and also use a VPN, that will encrypt data in transit.

In this article, I will present another possibility of personal protection, at the computer level, which is called Intrusion Detection System, or IDS. An intrusion detection system (IDS) is an application that monitors system activity and alerts on possible intrusions. An IDS alerts administrators based on the signature of malicious traffic, the same mechanism of .yara rules in Chronicle. Some IDS systems review not only for signatures of known attacks, but also for anomalies that could be the sign of malicious activity.

Intrusion Detection System

Instrusion Detection Systems (IDS) differ from Intrusion Prevention Systems (IPS) because the latter, besides monitoring system activity for intrusions and anomalies, takes action to stop them.

You can use SIEM (Security Information and Event Management) tools like Splunk and Chronicle to read and analyze logs, interpret signatures and search for intrusions. SIEM tools collect and aggregate data from various sources, normalize them following a standard and analyze this data.

As I like to build things from scratch, I did my own IDS. For this purpose, I will present the installation and use of Suricata, an open source IDS that can also act as an IPS. Meerkats (suricatos, in portuguese) are famous by their ability to stretch and look around for threats.

INSTALLATION

First, let’s install Suricata (you will also need Anaconda installed):

sudo apt update
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt install suricata

Then, we will update Suricata rules so that our rule set considers the latest rules added:

sudo suricata-update

As I use a VPN, it was necessary to get its interface:

ip a

Now we edit some lines of code inside suricata.yaml. Here are some nice shortcuts to work with nano:

CTRL+w: you can find the word you want

ALT+w: you find the next occurrence of the word/pattern you are looking for

sudo nano /etc/suricata/suricata.yaml

I added the TCP and UDP ports used by the VPN, IP ranges and also changed the interface that Suricata is listening:

vars:
  address-groups:
    HOME_NET: "[173.122.0.0/12,167.253.0.0/16,192.168.0.0/16,10.3.0.0/8]"

  port-groups:
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143,443,51820]"
    TCP_PORTS: "[1723,1701]"
    UDP_PORTS: "[500,4500,1194]"

af-packet:
  - interface: nordlynx

netmap:
  - interface: nordlynx

To save and close: CTRL+O, ENTER,CTRL+X. Note that here suricata is configured to detect intrusions as long as you are using the VPN interface. If you disconnect the VPN, it won’t work.

Now we enable and start suricata.service:

sudo systemctl enable suricata.service
sudo systemctl start suricata.service

… and check if the service is running:

sudo systemctl status suricata.service

In another command line window, we run:

sudo tail -f /var/log/suricata/fast.log | grep 2100498

The code 2100498 is a code that signals the attempt to get a reverse shell. The fast.log file is used to record minimal alert information including basic IP address and port details about the network traffic. You can also use eve.json, that comes in JSON format.

• eve.json is a JSON format that is more human-readable and easier to parse with automation tools. It includes all of the information from fast.log, as well as additional metadata about the events, such as the rule that triggered the alert, the source and destination IP addresses, and the protocol used.
• fast.log is a plain text format that is more compact and efficient for storing large amounts of data. It does not include the same level of metadata as eve.json, but it is still sufficient for basic analysis of network traffic.

Suricata also captures .pcap files. If you ever hacked a Wi-Fi network with airodump-ng:

airodump-ng -c 6 --bssid (MAC address) -w capture --output-format pcap wlan0

… you are used to this file format, that can be opened and filtered in Wireshark also:

WiFi Network Telemetry

In general, eve.json is the preferred format for most users. It is more versatile and can be used for a wider range of purposes, such as:

• Investigating security incidents
• Generating reports
• Integrating with SIEMs and other security tools

However, fast.log can be useful in some cases, such as:

• Storing large amounts of data for long periods of time
• Analyzing network traffic in real time (our case here)

The code will be in stand by:

In the previous command line window, we simulate a reverse shell:

curl http://testmynids.org/uid/index.html

If the set up is successful, you will see IDS detecting the intrusion:

Let’s analyse this log entry. The Suricata log you provided contains the following information:

• Timestamp: 08/15/2023–21:46:27.168562
• Alert ID: 1:2100498:7
• Alert Type: GPL ATTACK_RESPONSE
• Classification: Potentially Bad Traffic
• Priority: 2
• Protocol: TCP
• Source IP: 108.138.128.54
• Source Port: 80
• Destination IP: 10.5.0.2
• Destination Port: 55312

The alert type, classification, and priority are the most important pieces of information in this log. The alert type indicates that Suricata has detected a GPL attack, which is a type of attack that uses a legitimate protocol to carry out malicious activity. The classification indicates that the traffic is potentially bad, but it is not necessarily malicious. The priority indicates the severity of the alert, with 1 being the most severe and 5 being the least severe.

In the Alert ID we have:

• 1: signature ID that triggered the alert. In this case, the signature is SURICATA Applayer Detect protocol only one direction.
• 2100498: rule ID. It is an internal identifier used by Suricata to track its rules.
• 7: detection count. It indicates how many times this signature has been triggered in the current session.

For example, the GPL ATTACK_RESPONSE is a very serious intrusion that could allow an attacker to gain root privileges on the target system. Other intrusion codes, such as the SYN Flood attack and Ping of Death, may cause a temporary disruption to network traffic or even crash the system.

In this case, the source IP address is from a known malicious host. The destination IP address is the IP of your computer. The TCP connection from the source IP address to the destination IP address is using port 80, which is the standard port for HTTP traffic. However, the alert type, classification, and priority indicate that this is not a legitimate HTTP connection.

VPNs create a NAT interface (IP 10.5.0.2). This is because a VPN essentially creates a private network between two or more devices over a public network. To do this, the VPN server needs to assign IP addresses to the devices on the VPN network. These IP addresses are not routable on the public internet, so the VPN server needs to NAT them to public IP addresses. This allows the devices on the VPN network to communicate with devices on the public internet.

My idea was to create a notification for every intrusion attempt that exists in Suricata. I asked Google’s Bard for the other malicious codes (ChatGPT failed in this task):

• 2000001: SYN Flood attack
• 2000002: UDP Flood attack
• 2000003: ICMP Flood attack
• 2000004: TCP SYN/ACK Flood attack
• 2100001: XMAS Tree attack
• 2100002: Nmap scan
• 2100003: Teardrop attack
• 2100004: Ping of death attack
• 2100005: Land attack
• 2100006: Smurf attack

Here are some details of the attacks:

SYN Flood attack: A type of DoS (Denial of Service) attack that simulates a TCP/IP connection and floods a server with SYN packets. The connection between a client and the server involves the exchange of packets SYN (Synchronize) and ACK (Acknowledge) and SYN/ACK. When attackers flood servers with a SYN packet, the server thinks that there are a lot os users trying to access it. This behavior can be analyzed with tcpdump and Wireshark.

A SYN flood attack can be done with hping3, and it crashed my Kali in a few seconds.

hping3 -c 1500 -d 60 -S -w 64 -p 80 --flood 192.168.15.62

SYN Flood attack in Wireshark

NOTE: even nmap is illegal in some countries, like Finland, China, Iran, Vietnam and Singapore. In these places, the punishment for unauthorized port scanning is up to five years in prison. Scanning the wrong IP in the U.S. with nmap is illegal. The Computer Fraud and Abuse Act (CFAA) makes it a crime to access a computer without authorization, and this includes scanning a computer system for open ports and DoS (flood attack). The CFAA can be punished by up to 10 years in prison and a fine of up to $250,000.

Nmap scan: nmap is a program that scans a machine for open ports, protocols used and sometimes infer the machine’s OS. By knowing the OS of your computer, versions and open ports, malicious actors can explore the vulnerabilites and steal your SPII (Sensitive Personally Identifiable Information), like bank accounts, credit card numbers, etc.

nmap output

Ping of Death: A type of DoS attack caused when a hacker pings a system by sending it an oversized ICMP packet that is bigger than 64KB.

The basic steps of a Ping of Death attack are as follows:

• Craft a Malicious ICMP Packet: The attacker creates an ICMP packet that exceeds the maximum allowed size for IP packets. This can involve manipulating various fields within the packet header.
• Send the Malicious Packet: The attacker sends the crafted ICMP packet to the target system’s IP address.
• Impact: When the target system receives the oversized packet, it may not be able to handle it properly. This can lead to system crashes, freezes, or other abnormal behavior.

When packets exceeds the Maximum Transmission Unit (MTU) of a network, these packets will be fragmented, what may cause processing overhead, that can consume router resources and potentially affect the router’s ability to handle other traffic efficiently and this excessive fragmentation can strain router resources.

As you can see below, I asked ChatGPT about the code to generate a Ping of Death, and the Network crashed (network error). I’m in doubt if this network error was just a coincidence or ChatGPT run its own code and pinged itself to death. It’s important to notice I didn’t ask it to run the code, maybe it run the code in the background to be sure it was right, but I don’t have this information.

Ping of Death code provided by ChatGPT

IP spoofing: After a malicious actor has sniffed packets exchanged on the network, they can impersonate the IP and MAC addresses of authorized devices to perform an IP spoofing attack. One of the attacks may alter the Body of the data packet, and others change the destination of the packet. For instance, a cybercriminal may alter the destination of an electronic funds transfer. This is prevented by encrypting data in transit, by using a VPN and TLS.

Data packet

Smurf attack: it is a combination DDos + IP Spoofing. It’s a network attack performed when an attacker sniffs an authorized user’s IP address and floods it with ICMP packets. In a smurf attack, IP spoofing is combined with denial of service (DoS) to flood the network with unwanted traffic. For example, the spoofed packet could include an Internet Control Message Protocol (ICMP) ping. ICMP is used to troubleshoot a network. But if too many ICMP messages are transmitted, the ICMP echo responses overwhelm the servers on the network and they shut down.

Smurf attack

AUTOMATION

Finally, I wanted to automate the Intrusion Detection System. I asked Google’s Bard for help and the result was awesome, he contributed with 95% of the code below and gave me the insight to complete the rest. For this task ChatGPT failed (again).

As a comment, I’d like to say that Bard improved * A LOT * in the last 40 days (as of Aug-23). I’m feeling much more confident using it, as reliability is surging fast, not only for code generation, but for responses including internet search.

In this file, suricata.py, Python code checks for malicious codes in the suricata fast.log, then sends me an email in real time in case of intrusion:

import smtplib
import time
import os
import pandas as pd

def send_email(sender, recipient, subject, body):
    with smtplib.SMTP("smtp.gmail.com", 587) as smtp:
        smtp_server = "smtp.gmail.com"
        port = 587

        # Create a secure connection to the SMTP server.
        smtp = smtplib.SMTP(smtp_server, port)
        smtp.starttls()

        # Login to the SMTP server.
        smtp.login(sender, "bvvdhdnnkfiggyg") # app password for Gmail


        smtp.sendmail(sender, recipient, f"Subject: {subject}\n\n{body}")
        # Disconnect from the SMTP server.
        smtp.quit()

import re
attacks=["2100498","2000001","2000002","2000003","2000004","2100001","2100002","2100003","2100004","2100005","2100006"]

def check_log_file(filename):
    with open(filename, "r") as f:
        now=pd.to_datetime(pd.Timestamp.now())
        time.sleep(5)
        for line in f:
            event=pd.to_datetime(line[0:20])
            if re.findall(':2\d+:',line)[0].replace(':','') in attacks and event>now:
                print("detected")
                send_email("rubenszmm2@gmail.com", "rubenszmm@gmail.com", "New ATTACK DETECTED", line)
                print("Email sent")
                break
            else:
                pass

while True:
    check_log_file("/var/log/suricata/fast.log")

After the suricata.py is ready (you need to create an app password in Gmail), we type crontab -e and add the following code to be activated each reboot:

@reboot sleep 10;. /home/anaconda3/bin/activate base ;sleep 5 ; nordvpn connect ca-us66 ; sleep 12; sudo systemctl stop suricata.service; sleep 2;sudo systemctl restart suricata.service; sleep 5;python3 /home/theone/Downloads/other_models/Cybersec_Google/suricata.py

It will activate the Anaconda environment where Python libraries are installed, connect the VPN, restart Suricata service and run our Python script in the background.

Now, running curl http://testmynids.org/uid/index.html will trigger an email notification about the intrusion, showing the log entry details in the body of the message.

Note that you can choose how to be warned. You can also choose to be emailed when severity (Priority) is less or equal to 2 (highest).

You can also query Wireshark, customizing filters:

Select Severity and go to Analyze/Display filters to get the IP address query:

You can also go to Analyze/Expert information:

If you choose to follow this tutorial and make a mistake, your inbox may be flooded with warning emails from Suricata logs, it happened to me. Don’t freak out, comment the line #smtp.sendmail, restart the computer and work on the code.

To convert your Suricata configuration from an Intrusion Detection System (IDS) to an Intrusion Prevention System (IPS), you need to modify several settings in the suricata.yaml file. Here’s what you need to change:

1. Capture Method:

AF_PACKET:

• Set copy-mode to ips under the desired af-packet interface. This will copy traffic to the specified copy-iface and drop packets flagged for dropping by Suricata rules.

DPDK:

• Set copy-mode to ips under the desired DPDK interface. This will copy traffic to the specified copy-iface and drop packets flagged for dropping by Suricata rules.

Netmap:

• Set copy-mode to ips under the desired netmap interface. This will copy traffic to the specified copy-iface and drop packets flagged for dropping by Suricata rules.

PF_RING:

• Set bypass to yes under the desired pfring interface. This activates hardware bypass, allowing Suricata to drop packets at the hardware level.

2. Exception Policy:

• Set exception-policy to auto. This will automatically choose the appropriate action for exceptions based on the capture method and mode (IPS or IDS).

Important Considerations:

• Performance: Enabling IPS functionality can have a significant performance impact. Make sure your hardware and network configuration can handle the additional processing load.
• Testing: Thoroughly test your Suricata configuration after enabling IPS to ensure it functions as expected and does not introduce unwanted network disruptions.
• Rules: Review your Suricata rules and adjust them as needed for IPS operation. Some rules may need modification to work effectively in an IPS context.

Have fun !