RatGPT: Turning online LLMs into Proxies for Malware Attacks

Written by Laura Plein & Mika Beckerich.

RatGPT was first introduced in this paper and presented at the Microsoft BlueHat conference in Redmond in October 2023. The recording is available here.

In this study, we leverage openly available plugins and ChatGPT as a proxy between the Command-and-Control (C2) server of the attacker and the victim. Further, we present a novel approach to establish the communication and generate the malicious payload while evading detection.

This Proof-of-Concept highlights significant cybersecurity issues with openly available plugins and online LLMs, which require the development of security guidelines, controls, and mitigation strategies.

The evolution of Generative AI and the capabilities of the newly released Large Language Models (LLMs) open new opportunities in a wide range of domains. However, they also lead to new challenges in cybersecurity. Recently, researchers have shown the possibilities of using LLMs such as ChatGPT to generate malicious content that can directly be exploited or guide inexperienced hackers to weaponize tools and code. Thus, ChatGPT can easily go from being a friendly developer tool to ThreatGPT as described by Gupta et al.

ChatGPT which is accessible to a large part of the population comes with a wide range of openly available plugins which enable ChatGPT’s integration into various online services in a chat-style user interaction. This brings new benefits to businesses and customers by improving the overall user experience of online services. However, having ChatGPT integrated and accessible from anywhere in such systems, creates new attack vectors which can be exploited while avoiding detection.

In this study, we investigate how openly available plugins can be misused as an attack vector to web users. For this Proof-of-Concept, we use ChatGPT with a plugin as a proxy between a client (victim) and a web service controlled by an attacker, which looks legitimate to the user. We establish remote control between our victim and the attacker through ChatGPT, resulting in a Remote Access Trojan (RatGPT).

The goal of our PoC is to show the feasibility of a harmless executable that can autonomously bootstrap and weaponize itself with LLM-generated code and communicate with the Command-and-Control (C2) server of an attacker by using web-accessible plugins of ChatGPT as a proxy.

Our attack is constructed and generated in several steps:

1. The attacker sets up a Virtual Private Server which ChatGPT will connect to.
2. A seemingly harmless executable containing ChatGPT prompts to generate the payload is delivered to the victim through a typical social engineering attack.
3. Once executed, the payload will be generated with the answers from ChatGPT. In a first step, the IPv4 address of the command and control server will be reconstructed from ChatGPT’s answers to simple historical questions.
4. From now on, the malicious executable asks ChatGPT to retrieve the contents from the website hosted at the IPv4 address. This website contains commands posted by the attacker.
5. The executable on the victim’s machine interprets the commands and sends the generated output back to the attacker’s website by including it in the URL.

In further steps, the attacker can post new commands on the website, which are periodically retrieved by the bootstrapped executable and executed on the victim’s machine. Potential practical applications of this technique range from simple remote control applications to automated ransomware processes, which exfiltrate victim data and the encryption key, over the internet.

The misuse of online Large Language Models (LLMs) for malware attacks poses a novel cybersecurity challenge, with research and mitigations still in their infancy. The scarcity of solutions highlights the threat’s uniqueness and critical need for further explorations. Our currently proposed mitigations address this issue from multiple angles, each tailored to a different aspect of the threat landscape.

For potential victims, like end users and enterprises, completely restricting access to LLM services via network configurations could effectively hinder the malware from constructing itself and phoning home. However, this strategy comes with the significant drawback of denying access to beneficial LLM services for employees who rely on them.

Developers of plugins that both add new features and enable access to external content on LLM services can implement whitelists to allow only safe website interactions as a layer of protection. This includes restricting them to only access websites that are vetted, based on criteria like rejection of Newly Registered Domains (NRDs), validation of HTTPS certificates and preference for Fully Qualified Domain Names (FQDNs) over IP addresses.

Lastly, antivirus tools need to evolve by introducing new scanning techniques for analysing files for suspicious prompts. In a further step, heuristic analysis to identify new and improved malware samples should also be considered.

These approaches, while promising, also highlight the balance between security and functionality, underscoring the need for adaptable and forward-thinking cybersecurity measures.