Detecting Multithreaded Exfiltration in Zeek

Published in

Salesforce Engineering

9 min readMar 9, 2022

In a stereotypical heist, the thief’s “grab-and-go” plan often takes one of two forms. In the first, the thief grabs all they can and makes a loud and fast exit with explosives visible from miles away. In the second, the thief slowly and quietly amasses a fortune by stealing a few dollars at a time. It happens so slowly and subtly that only the most astute guards ever notice. What if a heist can be executed with both speed and stealth? You can imagine my surprise when I discovered that a thief in cyberspace could smuggle valuable data out of a network with both speed and stealth by utilizing a technique invented in 1968.

Data exfiltration (or exfil for short) is the unauthorized movement of data outside of a network. When an organization is attacked by malicious actors in cyberspace, stealing and exfiltrating data — business plans, intellectual property, or customer information — is often their goal. Multithreading is a common technique in a computer’s central processing unit that divides a large task into many smaller parts called threads. A task’s threads are executed in parallel, greatly reducing the overall runtime of the task. When network exfiltration leverages the concurrency of multithreading, threat actors can evade many traditional network detection mechanisms.

Additionally, there is a decreasing barrier to entry for multithreading as it is supported by many programming languages (such as Python) and can be implemented with only a few additional lines of code. Thus, the Salesforce security team designed a simple method for detecting multithreaded exfiltration. Implemented as a module for the Zeek (formerly Bro) network security monitor, the script generates a notification when exfiltration via multithreading occurs.

We have open sourced the Zeek script for detecting multithreaded exfiltration here: https://github.com/salesforce/multithreaded-exfil-detection/

Intro to Multithreading

If you are not familiar with multiprogramming concepts, here is a quick review:

A thread is a sequence of instructions within a program that can be executed independently of other code. For simplicity, you can assume that a thread is simply a subset of a process. Multithreading is defined as the ability of a processor to execute multiple threads concurrently.

In a simple, single-core CPU, multithreading is achieved by frequently switching between threads. This is called “context switching.” In context switching, the state of a thread is saved and the state of another thread is loaded whenever an interrupt occurs. Context switching takes place so frequently that all the threads appear to be running in parallel.

Threading can also be applied to networking tasks, including exfiltration. When threading is used to send data over a network, multiple connections and data streams concurrently send data over the network.

Exfiltration — The Story so Far

To understand why multithreaded exfiltration might evade standard detection logic, it is necessary to give a brief overview of the history of the hunt for network-based exfiltration. When a threat actor’s goal is to exfiltrate data out of a corporate network to their command-and-control (C2) server, there are two main options: fast and loud or slow and stealthy.

To use the analogy from earlier, “fast and loud” network exfiltration is like a thief that leaves a bank vault by blasting a hole in the wall with explosives and quickly escapes in a getaway car. When a threat actor tries to exfiltrate “fast,” it typically takes the form of massive file transfers leaving the internal network. These massive file transfers are “loud” because they are easily detected. When a massive file transfer occurs, the outbound byte rate and byte count of the network connection rapidly increases — a clear indication of exfiltration activity. If the exfiltration is successful, the threat actor steals a lot of data in a single fell swoop, leaving incident responders with very few options since their information is already stolen and the damage is already done. However, the exfiltration attempt is often so blatantly obvious that it will alert security analysts to the threat.

In contrast, “slow and stealthy” network exfiltration is comparable to the thief that quietly steals a dollar here and a dollar there without anyone noticing that money is missing. This kind of exfiltration is “slow” because threat actors exfiltrate data to their C2 server over weeks or months. Why does it take such a long time to exfiltrate? In order to be “stealthy,” the exfiltration of data blends in with the noise of normal business operations. By exfiltrating a few bytes at a time over a long period of time, security analysts are unlikely to notice anything anomalous in their network logs.

Figure 1: This is a visualization of the traditional exfiltration methods. The thickness of the arrow represents the amount of data being transmitted and the length of the arrow represents the time.

Multithreaded Exfiltration

Now that both multithreading and exfiltration have been introduced, it is time to see what happens when data exfiltration utilizes multithreading. Multithreaded exfiltration not only has the short time duration of “fast and loud” exfiltration but also evades detection more often than not. Network detection of fast and loud exfil often relies on a fundamental assumption: the exfiltrated information is sent over a single connection. Under this assumption, an alert will trigger if any single data stream meets a particular byte rate or threshold of exfiltrated bytes. However, when threading is used in data transfers over the network, many data streams — each a single thread — transmit a small portion of the data to the attacker’s C2 server. Individually, none of these data streams will raise any alerts as each one only carries a few bytes. Unfortunately, most network detection logic is not designed to aggregate numerous data streams to see a larger picture. Moreover, constantly tracking every data stream will tax the resources of any enterprise network detection system. Ultimately, however, the detection logic that only looks for a large burst of data in a single stream fails to detect multithreaded exfiltration.

Figure 2: A comparison of fast and loud exfil with multithreaded exfil. The single massive data stream (represented by the large arrow) in the fast and loud exfil diagram is noticeable enough to raise an alert. In multithreaded exfil, no individual data stream (represented by the small arrows) is notable enough to raise an alert, even though the same amount of data is being exfiltrated.

In addition to fooling traditional threshold-based detection, multithreaded exfil can also fly under the radar of the producer-consumer ratio (PCR). Devised by Carter Bullard and John Gerth, PCR was introduced to the world through their FloCon presentation. Formally, the producer-consumer ratio is defined as “A normalized value indicating directionality of application information transfer, independent of data load or rate.” Simply put, the PCR is a value between -1 and 1. If a device sends as much data as it receives, it has a PCR of 0. If the PCR is negative, this indicates that the device has downloaded more data than it has uploaded. If the PCR is a positive value, more data was sent than received. Thus, if a device is exfiltrating large amounts of data, its ratio should be more positive than usual.

However, when testing multithreaded exfiltration over TLS, the PCR did not immediately skew to positive values. In fact, the overall PCR of the multithreaded exfiltration connections was negative! This was the case for two reasons. First, in this testing scenario, only 530 bytes of data were sent per packet. Second, every connection required a TLS handshake that included the C2 server sending a large TLS certificate. As a result, each connection had a PCR of -0.334184, indicating that more bytes were received than sent. This did not match the assumed attack scenario where the exfiltration connection has a positive PCR value. Thus, multithreaded exfiltration was able to go undetected by PCR values.

Trying to accurately detect and alert upon multithreaded exfil with current solutions is hard. Solutions such as Netflow collection can theoretically provide the logs and visibility necessary for discovering multithreaded exfiltration attempts. However, a Netflow-based solution would still require security analysts to sift through the logs to find the exfiltration.

Detection Strategy

Based on our analysis, multithreading over the network can be detected in one of two ways:

Option 1: Calculate the ratio of outbound data to the number of connections. We expect a multithreaded upload to have many connections with a small amount of data in each. Thus, if the ratio is a high number, then it’s probably not a multithreaded upload. Whereas if the ratio is a low number, then it is more likely to be a multithreaded upload.

Option 2: Track the number of connections between a source and destination address with unique source ports. A multithreaded upload should utilize numerous connections as it sends data over multiple connections.

While both are theoretically viable options, we chose to go with Option 2 and track when multiple connections are seen with the same source IP address, destination IP address, and destination port, but different source ports.

We built our multithreaded exfiltration detection by modifying the exfil framework made by Reservoir Labs. The exfil framework is a suite of Zeek scripts that detect file uploads in TCP connections, including TCP sessions that have encrypted payloads. The script tracks every established TCP connection to determine if exfiltration is occurring. To detect multithreaded exfil, we added a table that tracks whether multiple connections have the same source IP, destination IP, and destination port. We also added an event and a function that aggregate the total number of exfiltrated bytes from the various exfil connections. If both the minimum number of connections and byte count are surpassed, then an alert is written to Zeek’s notice log.

When it comes to setting thresholds for exfiltrated data bytes, there is no one-size-fits-all approach, as environments vary. Fortunately, both the minimum number of bytes and the minimum number of connections can be easily modified in Zeek scripts. Ultimately, the detection logic is nothing more than a simple aggregation of all the threads. When testing the script against packet captures of multithreaded exfiltration, the script successfully raised a notice when the byte count threshold was surpassed, even though the bytes came from over 300 different data streams. Thus, we were able to accurately detect when exfiltration via multithreading had occurred on our network.

Of course, no detection tool is a silver bullet. In a production environment, the alerts from the exfiltration detection logic ought to be correlated with other log sources based on the connection parameters.

Testing

The script was tested on production sensors and over a trial run of 10 days. During this window, notices were raised when one of the machines in our environment made large outbound transmissions over multiple data streams. This indicated that the script was working as intended, detecting multiple streams of exchange between two systems.

During this time, resource utilization and performance metrics were also actively tracked. Despite our initial concerns, there was no abnormal behavior in the performance of our Zeek sensors.

Conclusion

Detecting exfiltration is a game of cat and mouse. Threat actors will continually find new techniques for smuggling data out of networks and security teams will find new ways of detecting them. While exfiltration leveraging a technique more than half-a-century old may evade traditional detection logic, we were able to improve detection scripts on our network monitoring sensors to help address this new play. As attackers implement new methods of exfiltration, detection products and security teams must match their pace.

We encourage you to experiment with our exfil scripts on your Zeek sensors: https://github.com/salesforce/multithreaded-exfil-detection/