The Greatest Step Change in Cybersecurity Ever!
Welcome to the New and Scary World of Generative AI and Cybersecurity
This is Day 0 of a new world of cybersecurity. Everything changes from here.
Introduction
There will be a time before Generative AI (GenAI) in cybersecurity and a time after it. Over the last two years, GenAI has come on leaps and bounds, and where it once suffered from hallucinations, took racist and bigoted approaches, and often was over-assertive, within ChatGPT 4.5, we see the rise of a friendly and slightly submissive agent, and that is eager to learn from us. This LLM (Large Language Model) approach thus starts to break down the barriers between humans and computers and brings the opportunity to gain access to a new world of knowledge, but, in the wrong hands, it will bring many threats to our current world.
There will be few areas, though, that will be affected more by the rise of Gen AI than cybersecurity. Why? Because the minute our adversories use it, we are in trouble. The hacking tools and methods of the past will soon look like the Morris Worm of the past. The threat landscape will see the rise of superintelligence and in providing ways for adversories to continually probe defences and gain a foothold.
Continual focus on a mission
These AI agents will not tire of their goal but continually focus on completing their mission. It will be a world of “crime-in-a-box” and see the rise of super affiliate networks, which are not driven by humans with scripts but by AI. Everything from the mission definition to the final payment for a successful campaign could be automated and intelligently driven — without a human hand ever touching the campaign. It could be a licence to make lots of money for those who like making lots of money.
The future could see AI implementing every part of the Kill Chain and without the touch of any human hands. For example, a GenAI agent could probe the defences of an organisation, and get a user to fall for a targeted spear phishing email, and then log into the system. Next, it could gather as much data as it could get access to (such as email addresses, contact details, email contents, documents, and so on) and pass it on to other GenAI agents. Next, it could then digest all of the known compliance documents in the world, and match them against the documents gathered, and send a report to the company about all of the data compliance breaches that the company is liable for (sent through legal documents, of course), and ask for a settlement or to go to court. This could all be done for the sake of the privacy of users and customers. It is a scary world!
The new threat landscade
Now a new report on Lloyds shines a light on this changing world of cybersecurity [here]:
It outlines that many new models have been released, and which focus explicitly on the creation of objectional material. Along with this, we will see the rise of improved cyber risk detection, such as through vulnerability discovery, campaign planning and execution, risk-reward analysis (with reduced costs of attacks for adversories), and single points-of-failure (as cyber defences become ever more dependent on LLMs). In the worst case, this could lead to a rise in cyber catastrophes and generate a new threat landscape (especially around nation-state activity and the low-cost barrier for entry for adversories).
The developed risk model in the paper takes of the four drivers of cyber threats (vulnerability discovery, campaign planning and execution, risk-reward analysis, and single point-of-failure:
We can see that vulnerabilty analysis and campain planning and execution have the highest levels of evidence in deployment, and also have the highest levels of potential impact.
The report outlines that from the classic 2017 paper from Google on the Transformer algorithm, we have seen an almost exponential advancement towards OpenAI’s GPT-4 and Google Bard, and the open sourcing of Meta’s GenAI models:
Vulnerability Analysis
The days of our quant old Pen Testing methods will quickly fade with the advance of GenAI. With this, LLMs will be able to quickly identify exploitable programming errors — and without the need for costly zero-day analysis. The asymmetric of attacks will become increasenly apparent, as software vendors will struggle to keep up with cybersecurity experts using GenAI methods. While the less experience actors will possibly be detected with the new generation of vendor tools, the skilled actors are likely to easily bypass existing controls — even if AI generated.
These vulnerability can then be easily shared with other evil GenAI agents, and where this scanning can be easily automated and become distributed. They will then be able to deliver embedded micro-code and firmware; decompile binary files for executable; and modify device drivers. Overall, the GenAI agent will have a whole box of tricks that it can inteligently deploy against a target network and go through the stages of survalience, gaining a foothold, traversing the system, and then creating the end impact (typically either encrypting files or exflitracting data — or both). But, in the hands of true cybersecurity experts, these tools could be absolutely devistating.
Along with scanning, we could see LLM-powered malware run on small devices, such as Raspberry PI. There are already products appearing on the market which advertise AI-driven stealth techniques that run on the R-PI. These devices would be used in areas where physical access would be required, such as gaining access to wi-fi networks, or targetting specific hardware devices. Thus, physical hardware could be at a greater risk with the rise of GenAI, and where companies need to scan for planted hardware devices.
Campaign planning and execution
If you think that ransomware is bad now, then just wait for the GenAI-enabled ransomware era. These will involve intelligence agents which could scan for targets, define their key characteristics, and automatically gather data on them. From there, they could then gather the right resources required to target the networks and then conduct broad and/or targeted phishing campaigns for very little financial cost. This would even be a disinformation campaign against a company and where bad news spreads on social media. Every single person in a organisation could be a target, too. At the core of this is likely to be the exfiltration of data, which could damage a company’s reputation or lead to fines. But GenAI could go one step further and actually analyse potential non-compliance issues from the data, and threaten the company through the courts and class actions.
The GenAI tool will have many modes of transport for the attacks or deformation, including the automated synthesis of phishing, impersonation, and defamatory materials. It will be a world where it will be difficult to tell fact from fiction, and the days of spotting a spearing phishing email with bad grammar will recede into the past.
AI Governance
This rapid rise has seen the UN and the EU setup governance policies for the general use of AI, but it is unlikely that our adversories will follow these approaches, and will build AI for profit. The EU has taken a ‘risk-based, top-down legislative approach’, while the UK has taken a principles-driven approach to the use of AI. When it comes to “AI Safety”, the report outlines that there should be guard rails in the areas of:
- Autonomy and advanced capabilities. This focuses on whether there could be a risk to the general public in the implementation of the AI methods.
- Content generation of the models. This focuses on the data produced by the AI methods, especially related to privacy breaches, disinformation, copyright reaches, and so on.
- Malious use of the models. This focuses on the damage to people and their property — either physical or virtual assets.
For governance, the report outlines that the weights used in the models that were created from training should be kept private and not released to the public. This will stop threat actors from copying the engine and re-enforcing it with malicious data. The model training should also be setup in an isolated environment and should not be made public. Along with this, there should be strict rules placed on data quality, ethics, privacy and accountability, and these apply for the whole life cycle of the AI entity.
Financial and computational costings
It is well known that training and supporting LLMs can be a costly business. The report quotes that it cost Meta 3.3 million hours of computation for the creation of Llama2 — and which cost around $10 million for electricity and hardware. The Llama2 model and its weights are available to the general public under licence terms. The vast costs it takes to train these models, though, are likely to limit the current scope to the main LLM engines to OpenAI, Meta, Anthropic, and Google.
The EU and other regions of the world are aiming to constrain the growth of AI, by making it more accountable, more trusted and with verifiable moral and ethical approaches. But advancements have been made in LLM training, and where it can now be run on commodity hardware, such as in running ChatGPT 3.5 locally on a Macbook with an M2 processor (without the need for any Internet connection).
This type of approach will bypass all the safeguards, and where models can be updated to add malicious entities. At the fingertips of virtually every person on the planet, we have (possibly) the ultimate set of hacking tools that could breach more organisations.
Guard rails
In order for these AI entities not to be used for malicious purposes, there must be strict limits placed on the usage of input data, use adversarial testing (“a Pen Test” for AI), and have limits placed on user interface accesses. Increasingly, ChatGPT will not return a narrative if it thinks that the request is immoral and unethical.Overall, LLMs are learning to take our point of which they can learn. In this case, it takes my point on North Korea showing tendencies towards a 1984 world:
There are though many ways to “hack the console”, and overcome the limits on performing bad actions:
Conclusions
The speed of development of LLMs has caught our lawmakers unaware, and now they are perhaps trying to put the genie back in its bottle. The bottle, though, perhaps gone and the genie is nowhere to be seen. While many will peddle the usage of LLMs in medical and smart car applications, there will be a whole lot more who will see dollar signs in ransomware attacks. Those with the greatest skill levels will be able to use GenAI for purposes that they have never seen before and which could have catastrophic effects.
We can aim to constrain the advancement of good GenAI, but bad actors will increasingly use it within unconstrained applications. The release of Sora will see a rise in almost real-life video fakes and open the door to many new threats. If you are interested, here is our review of the area:
My advice to organisations? Invest now in recruiting and keeping the best cybersecurity talent that you can find, and keep them. On the one hand, they can build the AI-driven defence tools that your company needs, but on the other, they will understand the realm of the possible. The GenAI box cannot be closed, so we need to understand how it will shape our new world of cybersecurity. Otherwise, we risk cyber catastrophe.
