How to Test Your Employees With a Phishing Simulation Campaign

Craig Hays
Jan 29 · 14 min read

Running a successful phishing simulation campaign is difficult. This is how to test your employees to better prepare them for real attacks.

Fishing Fleet image by Bernard Benedict Hemy

Following on from my last article on how to write phishing emails that work, this article outlines how to run a successful phishing simulation campaign. We do this to measure how employees respond to attacks from criminals, how effective our training is, and where we need to do more to help our employees stay safe online.

Who Should Be In Scope for Phishing Simulation Tests?

As always, the answer is everyone. We’re all susceptible to phishing attacks. I might be a bad example but I’ve clicked on links in tests I’ve written, scheduled to send, and then forgotten about. When you’re caught up in your daily work it’s easy to get caught up in the whirlwind, not thinking about what you’re doing until it’s too late.

My preference is to test everyone on a loosely scheduled basis. If it’s always the 1st of the month people will expect it. Throw some randomness in there and aim to test everyone at least every 30 days with the same test.

In addition to this, I like to target high-value teams and individuals for further testing. Teams like HR, Payroll, Finance, and IT to name a few. For individuals, the CEO, COO, CFO, and directors of each department are good choices. I focus on people who are likely to be targeted by actual criminals. People who offer the biggest possible reward if their accounts are compromised.

Often, senior people like the CEO, COO, and CFO have assistants that read their email and manage their mailboxes for them. Their assistants are as vulnerable as the mailbox owners, if not more so. When we test senior individuals we’re also testing those they trust with access to their online resources. The impact of a compromise is the same regardless of who is sitting at the keyboard.

Types of Phishing Simulation Tests

There are many different phishing scenarios you can play out but they tend to fall into the following categories. Emails with:

  • A link to a malicious website
  • A malicious attachment
  • A benign attachment that contains a link to a malicious website
  • Neither a link nor an attachment but a request for the receiver to perform some other form of action. E.g., reply with sensitive information, make a payment, make a request to someone else

Malicious websites can be set up to capture login credentials. To do this they present the user with a login form when opened. In a very basic test, this could be a plain white web page with a simple web form for username, password, and a submit button. In a more advanced test, this could be a like-for-like copy of the website alluded to in the body of the phishing email. The more seamless, realistic, and familiar the experience is, the more likely you are to succeed in phishing your employees.

A malicious website could also be set up to deliver a malware payload. Some phishing simulation tools track which version of the browser, operating system, and plugins the client is using. In a real attack, a malicious website would use this information to deliver a payload that would execute on the client, exploit a vulnerability on the user’s device, then add the machine to a botnet, install ransomware, or worse. In our tests, we just want to know that a vulnerable machine connected and could have been exploited.

A malicious attachment could be anything that executes directly on the client when manually opened or executes automatically when downloaded by a mail client with unpatched vulnerabilities. The preview reading pane is an easy way to execute code without intending to. For our testing, we just need to know that the attachment was opened. Phishing simulation tools that allow you to include attachments usually include a benign mechanism to track when an attachment is opened and displayed to the user.

A benign attachment with a link to a malicious site could be a word document, a PDF file, an HTML page with a link in it, a spreadsheet with a link, etc. The list goes on and on. By including the link in an attachment it makes it easier for it to slip past our defence mechanisms. Once it reaches the end user’s mail reader it’s just a matter of using the right words to get them to open it.

This one is tricky as it’s hard to measure how many people perform actions out-of-band. If you ask someone to forward sensitive information somewhere it might not come from the email chain your test created. There may be a significant time delay from request to response. The conversation may need some form of back and forth between the employee and our ‘attacker’. These types of tests are more typical of spear phishing attacks against individuals in key positions within an organisation. We should still look to include them in scope but they’re a lot more complicated to manage. The potential risk associated with running these tests is also a lot higher as someone may do exactly what you ask without question. They could cause as much damage as a real attack!

Some of the feedback I received on the last article was around the ambiguity between carefully crafted phishing emails and Spear Phishing. To me, Spear Phishing is about targeting an individual using information gathered through Open Source Intelligence to give credibility to the attack. My standard phishing emails in comparison are sent to a demographic of thousands of people in a loosely related group. There is nothing specific to each individual in the email but the underlying psychology targets the emotions of the majority. Not everyone will take the bait. All I really need as an attacker is to fool one person enough to get my foot in the door.

What To Do Before Starting a Phishing Simulation Campaign

Before you start sending people phishing emails it is critical that you have buy-in from the senior people in your organisation. Phishing simulation emails are essential for raising the awareness of threats to people in your organisation but they’re not always nice. Effective emails create a strong emotional response in the minds of their readers. You never quite know what is going on in other people’s lives at any given moment and your email might be all it takes to push them over the edge that day, especially if they don’t realise what they’re reading isn’t real.

Without the support of your organisation’s leadership, you may be asked to stop testing, either temporarily or permanently. This isn’t good for your organisation as realistic training is essential for beating the real thing. If something bad does happen due to one of your tests, for example, someone walking out of the office or resigning, having buy-in from the top will reduce the impact on you as an individual. You may be asked to tone it down in future or warn specific people depending on the theme of the email you’re sending, but probably nothing more. Without buy-in you’re a rogue agent sending awful emails to people, telling them to log in to a website or you’ll slash their pay.

We also need to ensure our phishing emails reach our people’s mailboxes. To do this we need to whitelist the IP addresses of servers sending the emails and if possible create rules to ensure that all emails go straight into the primary inbox folder. Some clients will automatically move emails with certain keywords into the ‘other’ folder. People often create custom mailbox rules to do similar things. We need to ensure that these do not apply so that our test has the best chance of being seen.

There’s no point testing how people respond to phishing emails if you’ve never told them what they actually should do. Before you start testing, and on a regular basis afterwards, send out some training on what to do if you receive a suspicious email and what makes an email look suspicious. People will forget so this needs to be repeated on a scheduled basis.

There are a lot of options out there for phishing simulation testing. I’m not going to list them or make any recommendations. They all do pretty much the same thing. The important part is that you can measure and track key metrics over time, identify repeat offenders who need additional support, and provide and track the uptake of end-user training.

If you have a small budget you can opt for an open-source solution. If you have some money to spend you can use a proprietary system. The more you pay the less work you’ll need to do (generally). It’s all about finding balance.

What I’ve discovered with all of the tools I’ve assessed so far, the phishing emails that come as standard are generally poor. They are the generic spam that ‘fire and forget’ opportunist criminals are using to target every email account on earth in the hope that something sticks. The stock emails won’t let you run tests with the level of sophistication that comes with an attacker who’s willing to do the slightest bit of research on your company or who understands human psychology and the impact language has on invoking thoughtless responses. For this, you’ll either need to write your own or hire a specialized copywriter who can help target your people effectively. People who write real emails use copywriters, so why not use them for our fakes?

You can run your own phishing simulation campaign for a very reasonable price or you can outsource it to someone else to run it for you. If you outsource the whole process it’s going to be difficult to find someone who cares about the effectiveness of the testing as you do. If you do it all yourself you’ll need to pick up the administrative overhead of running the phishing platform. My preference is a combination of both: a managed service that someone else runs for me, keeping it running, generating monthly reports and insight, and letting me steer the content, schedule, and targets for testing while they do the legwork.

If you’re looking for phishing testing as purely a tick box exercise to satisfy some form of compliance requirement, outsource it and forget about it. Read the monthly reports and rest assured that you’re doing the bare minimum. If, however, you really care about the results of your phishing simulation and training efforts then you need to invest your own time and energy in the process. Nobody understands your business as you do. No one cares as much as you do. Nobody can make it as successful as you can with a combined team effort.

There is value in phishing simulation tests beyond the just-in-time training opportunities. By identifying and tracking key metrics we can measure how people are performing over time. We can identify where we need to improve our strategic training and prioritise our other cybersecurity improvements to meet the threats we face. The key metrics I measure and work with are:

  1. Theme of email
  2. Total emails sent
  3. Number of emails delivered
  4. Total emails opened and read
  5. Links clicked, attachments opened, etc.
  6. People who leaked sensitive information
  7. People who then completed the just-in-time training

With these measures, you can detect which type of phishing threats your people are most susceptible to and tweak future tests accordingly. Metrics 2–7 create a phish funnel, similar to an eCommerce funnel, that people move through on their phishing experience. Ideally, we want everyone to stop at point 4 but that doesn’t often happen. Anyone who gets to 5 or beyond repeatedly should be flagged for further support and training.

Running a Phishing Simulation Test

Timing and duration are important factors when running a phishing simulation test. We want phishing emails to land in inboxes when people are actively looking at them. The emails at the top of the pile are the ones that get addressed first. Those that aren’t touched in the first ten minutes to one hour may never be opened.

If you have a small group of users, say less than 100, sending all of the emails at the same time might be OK. If you have several thousand you’re going to create a lot of noise with a big bang approach. I prefer to trickle feed phishing emails to the entire organisation over several days with a hundred or so sending each hour. This reduces the amount of noise created and cuts down on the “hey, I have that email too!” chatter. If a room full of people get the same email the first person to get feedback is likely to tell everyone nearby. If you can delay the others receiving the test there’s more chance of getting them back into their usual state of awareness (or lack thereof) and therefore get a more accurate test result.

I tried A/B testing phishing emails in a single test to see which performs better but then realised that wasn’t the goal. When running a phishing simulation test you want all of your users to have the same experience so that you can assess them all under the same conditions. Running a 50/50 split makes it difficult to draw conclusions and determine what you need to do to improve things.

A single email to all employees allows you to create a standard measure across the organisation. Changing the email each test is fine, as long as everyone gets the same one. We’re trying to test our people’s response to threats, not our ability to write phishing emails.

A well-executed phishing test is going to create a lot of noise. Your IT team is going to receive a surge of suspicious email reports, which is great! They’ll still need to be assessed to prove they’re part of the test before being discounted as all suspicious email reports need to be taken seriously. This adds extra workload to an already stretched team. Ensure that they know your test is running, what the email looks like, who the sender address is, and work with them to minimise the impact on them.

If your email mentioned a problem with IT Services, your IT Service Desk function is going to start getting support tickets from people who believe the email is real but haven’t managed to follow the instructions in it. If your email relates to HR, the HR department is going to start getting concerned phone calls and emails. Emails related to payroll make noise for the payroll team, etc. It’s important that we run realistic tests. Real criminals pretend to be part of your company every day. By not replicating this behaviour we fail to prepare our people for the realities of cybercrime.

I mentioned earlier that this article is a follow on from my last post, 9 Things I Learned Writing Phishing Emails. Your phishing emails need to be as realistic and effective as those your employees will see in the real world, if not better. We use tools to block over 99% of the malicious emails that are targeting our users. Those that make it through are either going to be really obscure and ineffective, or very, very good.

If you run your tests with mediocre content all you will achieve is to create a false sense of security and the illusion of improvement as people get better at spotting and reporting bad phishing emails. When the real thing hits it will hit with devastating effect and your people will be unprepared. Make your training as close to real as possible so that genuine phishing attacks do not come as a shock to those who receive them.

Like Java’s just-in-time (JIT) compilation, just-in-time phishing simulation training lets me target people when they need it most, right after they’ve taken the bait and compromised themselves. After my testing scenario ends I display a just-in-time piece of training specialized to what just happened to them. This needs to be a bite-sized piece, suitable for consumption in a minute or two. Any longer and people will lose interest.

It’s easy to get caught up in providing training to those who fall into the negative bucket, the people who ‘failed’ our tests. However, it is equally as important to provide feedback to those who performed well. Anyone who reports a phishing simulation test email to IT Security should receive a ‘Well done!’ message that acknowledges their correct response to the presented threat and reinforces their behaviour as the correct one.

Some providers offer a button that attaches to the Outlook app to report suspected phishing emails. While this is great, it only supports people in the desktop environment. More and more people being caught out by phishing attacks are using mobile clients. Mobile users could be using one of any number of clients, depending on your organisation’s policy for reading emails on mobile devices. For this reason, I prefer the plain old ‘forward the email to IT Security’ approach as it is compatible with any device a user may use.

I configure the IT Security mailbox to watch for these reports and automatically respond with a ‘Congratulations!’ email when the reports arrive. You can add additional pieces of workflow at this point for real threats such as forwarding to a threat protection tool to purge similar emails from everyone’s mailbox, report it to Microsoft’s security team, and more, but for our tests, we simply need to respond with a positive, reinforcing message.

For my responses I use this wording similar to this:

Congratulations! You passed this test!

Thank you for reporting this suspicious email. This time the threat wasn’t real, but that isn’t always the case.

Phishing attacks just like this one happen every day. Real criminals are out there and they’re trying to steal your password, empty your bank account, read your emails, and take over your computer and phone. We send phishing tests like this to everyone on a regular basis to measure how we all respond to attacks from criminals, how effective our training is, and where we need to do more to help you stay safe online.

Please keep reporting suspicious emails like this one to <it-security-email-address>. We’re doing everything we can to stop them reaching you but we can’t catch them all and we need your help. The more you report to us the better we get.

Thanks again! IT Security

The final category of people is those who take no action. They neither act upon the contents of the email or forward it to IT Security. It either just sits in their mailbox forever or is deleted. I haven’t worked out what to do for these people yet. If you have any thoughts please let me know.

The first few tests you run will provide interesting results but nothing more. It’s only once you get to around 6 tests that you can start to derive actionable insights from the results. People who repeatedly fall for your phishing tests should be prioritised in your deployment of Multi-Factor Authentication (MFA) if you haven’t already deployed it to all users. These are people who consistently give away their login details to phishing sites and are a huge threat to your organisation.

You should also create additional training for repeat offenders to help them understand why phishing is a serious problem and what they can do to help us spot it, report it, and stop attackers from getting to us. If additional training and support is ineffective then escalation to their managers should be considered but it’s tough to agree on what actions to take beyond blocking external access to their account until performance improves.

Summary: What a Succesful Phishing Simulation Campaign Looks Like

A well-executed phishing campaign will send fake phishing emails to employees on a regular basis to test their responses. Anyone who takes the bait will be given immediate feedback on what went wrong and how they can do better in future. People who identify the phish will either take no action or report the suspicious email to your IT Security team. Those who report it will receive immediate, positive feedback to reinforce this behaviour. Results will be measured over time and used to guide your cybersecurity strategy and roadmap.


Originally published at https://craighays.com on January 29, 2020.

Cybersecurity Operations

Cybersecurity Operations

Craig Hays

Written by

Aspiring writer, Cybersecurity Architect, Bug Bounty Hunter, Musician, Movie Producer, Failed Skydiver.

Cybersecurity Operations

Cybersecurity Operations

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade