How to build an effective red team

This post is a collaboration between myself and Samantha Davison, Trust Engineering Leader. Sam is an expert in transforming security culture at large organizations including Snap, Lyft, and Uber, etc. and has contributed her in depth research to the cultural change section of this piece.

After being immersed in the red team community for 5 years, leading red team operations, and spinning up my own red team I’ve acquired some knowledge on the best ways to run a successful red team, and more importantly, how you can completely mess it up.

Trust is hard to build and easy to destroy.

In order to tell you how to be effective though I have to start out with how I define a successful red team. That definition may come as a surprise, and my aim is to help you consider how you structure your team and the work they do.

I shared my knowledge on how to build effective teams with the community last year when I spoke at the Australian Women’s Security Network’s Women in Security conference in Melbourne Australia with my talk titled “Maximising red team effectiveness by hacking with transparency & consent.”

Since the talk was well received I’m sharing a written version here as it wasn’t recorded. This post is for anyone building a red team, thinking about starting one or trying to reform one that has fallen out of grace.

Table of Contents

What is a red team?

• Explanation of the real purpose of red teams.

How does the red team add value?

• The value you get from red teams, and I reveal the secret extra spicy good value that most people fail to get when they red team.

Why does it go wrong?

• Why people fail to get this value from red team operations.

How can you maximize chances of capturing value?

• Methods and strategies I’ve devised for making sure you maximize the value your team can give.

What is it to be a red team?

The age-old debate among red teamers everywhere. To keep it simple today I will define “red team” like this:

We act like hackers, we hack into your company, we tell you about it.

This article focuses on a very specific type of red team project. These projects (usually called ‘operations’) are typically large scale, high-impact, and as realistic as possible to a real-world breach scenario.

To be clear this is by no means the only thing a red team can, or even should do.

The best red teams are ones that are driven by the needs of the organization.

They can adapt to the maturity level of the security program they are working with and can modify their methods and strategies to keep a smooth cadence of operational improvement from the defensive team following their efforts.

For the context of this post we are discussing one typical service offering of red teams: the big impact, hard-hitting style of operations that I have the most experience with. For simplicity, I’ll call them “Full-scope ops”.

Full-scope ops have the following properties:

1. Based on Analysis of Recent Attacks

What would be the attacker’s motivation? Mimic tools and techniques. These attacks aim to be as impactful and realistic as possible, within reason.

2. Limited Internal Knowledge

Show it could have been done by a ‘real’ attacker. Because my red teams have been internal to the company we go to lengths to show that our attacks could be executed by someone with no internal knowledge of the company. This lends credibility to the final result.

3. Delayed Red Team Reveal

Knowing it’s a test changes the response. Delay if possible.

To get the most out of incident response, delay telling the responders that the exercise is a red team one, and let it play out as far as possible until the game moderator (GM) decides to reveal the truth.

In case you are wondering, the game moderator in a red team operation is typically the security team manager or someone who can oversee the operation with full visibility. It is their job to ensure that the operation stays within acceptable boundaries, the people involved in red and blue teams are happy and engaged, and can call an absolute stop if anything goes wrong.

This is negotiable and deciding to do this depends on whether you plan to test Incident Response capabilities during the operation and whether the incident response team is on board with and prepared for this type of no-reveal situation. As I’ll discuss later, proactive sourcing of approval and agreement is very important to the success of the exercise.

How does the red team add value?

The red team emulates attackers who intend to cause harm to the company. At the end of the day though our real purpose is the same as the rest of the security team: improving the security posture of the company.

Let’s break down how the red team does that.

There are three main ways a red team can bring value to a company.

1. Finding vulnerabilities that exist and reporting them to be fixed.

People think this is the most important output, but in my opinion it is a symptom rather than the goal. It is also the most straight forward. On the way towards their ultimate objective the red team will inevitably illuminate holes, vulnerabilities, bugs, crashes, and whatever other loopholes somehow overlooked beforehand.

The red team doesn’t find vulnerabilities as their primary mission, however the ones that are exploited while mimicking a real-world threat are often seen with renewed criticality, as their relevance can easily be tied to an direct business loss.

2. Exercising the Incident Response process

Using a red team for full Incident Response simulation is optional. The reason it works so well is that unlike a real breach where at the end of the day you may be left in the dark to large parts of what occurred, in a red team operation you can have full information of what happened at every stage of the attack. This allows for accurate measurement and understanding exactly where change is needed to improve prevention, detection, and response for the blue team.

3. Cultural Change

Positively influencing lasting security culture change is the sign of a truly effective red team.

Take a minute to think back. Remember the first major security incident you experienced in your professional career, and how you felt when it happened.

The questions asked in the aftermath: what can we do to make sure that it never happens again?

The feeling of the whole company suddenly understanding the urgency of security and putting in whatever effort they can to help.

What exactly are the behaviors that indicate cultural change? How do you measure it? Why do many red teams fail at this critical goal? One approach you can take is to focus on the behaviors that you most want employees to exhibit in the aftermath of your work.

If your security team has a behavioral engineering or security engagement group, they could be excellent partners to your red team for the cultural change components of your campaigns. They can help identify behaviors, include your stories in training, and often have a line to corporate communications to get your results maximum exposure.

When dreaming up your red team operations, consider the behaviors that might allow you to be successful. Is it an unpatched system? Insecurely written code? Overly permissive systems? A too-good-to-be-true phishing site? Holding the door open? Your operation may take you down different roads than you conceived at the start, no doubt. Note the behaviors you think you will be testing at the start and document the ones you discover along the way — these are real things you need to address post-op.

Cultural change is the cumulation of the positive (or negative) behaviors exhibited in your organization. A mature culture will be reinforced by the consistency of all of the behaviors below. An immature culture will have inconsistently positive behaviors or the absence of most if not all.

Sample Behaviors

How do I measure cultural change?

Pick your behaviors

First, key in on the top 3–5 behaviors that would be most indicative of positive/negative culture in your organization. Each company is different and what works for a Fortune 500 won’t necessarily fit with a startup. Treat this like reconnaissance. Read up on your company mission statement. Look at examples in other culture building teams. Get input from other security sub-teams. They will need to help normalize these behaviors in their work as well.

Perhaps most importantly, select behaviors that you have the tooling to measure or can easily build. You might feel up to manually measuring a couple of times but if you want to incrementally grow this program, you’ll want to automate this part as much as possible.

Baseline

Take your first measurement of your selected behaviors, a baseline, before you begin your red team activities. This will produce the purest metrics as you will understand the state of your culture before shaking up the snowglobe with your campaign.

Post Campaign

Take your next measurement after your campaign has concluded, leadership and impacted teams have been briefed, and you’ve evangelized the story to as much of the company as you can. This measurement will deliver early indicators of the cultural impact of your work.

Repeat on 3-month intervals following the post-campaign measurement. You may see spikes in one period or another. Maybe one behavior will take hold while the rest stay consistent. This will be the fun part, to understand what does or doesn’t stick and how you can influence your teammates to adopt better behaviors. Lasting cultural change takes time and you may need to pivot to get your full behavioral payload.

A full scope red team op can be the pivotal event that initiates the cultural change I’m talking about. Recreating events like that is where an enormous amount of red team value lies.

Why does it so often go wrong?

Red team incidents like the ones I’m describing are fraught with danger. When we initiate a red team incident we are essentially playing out the worst day of the lives of some people at the company and the security team. It is high stakes. There are so many ways this could go wrong.

I’ve seen it go wrong.

You might think that if you take a hacker

+

Point them at their target or objective

+

Grant them a license to hack

+

And let them create chaos

=

That as a result of having completed those actions cultural change will follow.

It’s not that simple. Make a mistake and you may lose the valuable cultural change you so badly want.

If you let a red team loose on the company without the appropriate planning and considerations you risk losing hard-earned good-will and political capital. You created an incident, you caused people stress, you surprised people unexpectedly, and now you expect them to do EXTRA work to fix the things you found, and no one even asked you to do it! Or worse yet, your boss asked you to do it but none of the people inconvenienced in the aftermath care about that.

If you embarrass people and catch them off guard you’ll make sure that the message you’re trying to send doesn’t make it far.

How do I maximize the chances of influencing cultural change?

I’ll let you in on a secret. The difference between a red team who can’t capture the elusive cultural change impact and one that can isn’t in their technical ability.

It’s all about how we communicate with the organization.

What follows are my three suggestions — the things that I’ve found work exceptionally well to preemptively mitigate red teaming going sideways and can guide you towards the goal of widespread security culture change at your organization. The first two are about timing and audience. The last one is about tone and how you craft your message.

How can I ensure my red team maximizes its value?

1. Consent based hacking

It starts before you even begin.

If you don’t get buy-in a red team operation can feel like you were blindsided and attacked. It can seem like a betrayal. It will cause distrust.

When I gave this presentation at the Women in Security Conference last year I told a story about an executive who hired a red team to shake up the status quo and prove their point about security at the company. After the talk I had someone come up to me and tell me a story about a CIO that hired a red team for similar reasons. He wanted to prove that a certain product group wasn’t doing a good enough job. He didn’t tell anyone about the test, and they found out about it at a meeting after it was over for the first time. The product manager and engineers were furious. The CIO had an immensely hard time getting them to cooperate afterward and it damaged their relationship with that team.

To combat situations like this occurring, the person who will be the highest level of stakeholder for the target of the test should always be asked: Would you like a red team engagement?

Always make sure that the person responsible for the roadmap and future planning is also included and that they understand that the outcomes of the test may require some setbacks to the schedule. And of course, the person who is responsible for the major outcomes of the team, usually C-level executives should be sponsoring and signing off on the test so that the people below them feel supported in making changes to roadmaps to make the security improvements that are needed.

What points should you cover when getting approval for your op?

Once you have asked nicely if they would like to be hacked and they have said Yes! You can delve into details.

• Write a scope document explaining the threat actor you will be emulating and what the expected outcomes might be.
• Describe in detail how you plan to communicate post-operation with the company so they aren’t surprised by what comes next. They should be anticipating the attack, and be ready to respond appropriately.
• Leadership should feel like they are in control of the situation, and it was their approval that led to the benefits of the red team operation. No one likes surprises, especially stressful ones.

What teams should I avoid targeting as part of the operation?

• Teams that are in the middle of large security improvement projects

Typically they already understand they have security issues to fix and have the momentum and buy-in needed. They don’t need the extra push.

• Teams who are being onboarded as part of a new acquisition and haven’t been integrated with the company security framework and tooling.

Their security posture will change once they are integrated and the findings now won’t be applicable in the future.

• Teams who have recently undergone testing and have findings to fix

2. Tell Everyone (if possible)

This one can be a hard sell. It’s scary, right?

Revealing how the system was flawed and admitting our mistakes. Too often the results of red team operations are locked down, restricted to upper management, and then fixes dictated out to employees with no explanation.

Although this can work, it degrades trust. People will suspect that something strange is going on. They will think:

1- What happened? And why doesn’t the company trust me enough to tell me about it?

2- This is silly, we are fine how we are and this is just security trying to force more bureaucracy into our lives.

You want to avoid that, you want people to see real reasons why they are making changes and feel like it is the right move. With the buy-in you got from stakeholders with step one, you should be in a good position to send out the message in a positive way to a wide audience.

Tips for telling a wide audience about your hacking success

Aim to entertain

• Write or present an entertaining version of events that can be consumable by everyone at the company.
• You might need more than 1 version- not everyone will care about the same things.
• Tech people need tech details + implementation, non-tech care about social + general best practices.

Common formats across red teams I’ve interviewed seem to be:

• 15-minute slide deck for the exec team
• Non-technical impact focused narrative blog/document for company-wide consumption
• Technical document with hacking techniques included and explanations of how they work for technical teams involved with fixes.

Focus on the impact

Focus on the impact the red team hack could have had if it had been real.

Remember how I asked you to think about the first big breach you experienced and how you can still remember it today?

The goal is to get that same sort of visibility and awareness across the company, but without having to experience a real breach. You can’t do that if you don’t spread the story far and wide.

Put on your marketing hat

This is an entertaining event and story. Promote it with logos, teaser emails, posters, videos, whatever you think will engage people in the story.

Timing is important

If you have critical findings that require urgent fixes, patch those first before telling a wider audience about it!

3. Serve humbly

Lots of people have written about why it’s important to not shame or blame when delivering test results. I think it’s important because I’ve personally witnessed the superior attitudes of hackers and red teamers causing drama and fighting inside security teams.

I’ve got a good story of where red vs blue clashes caused an unhealthy dynamic.

This red team was composed primarily of red-team-4-lifers who had always worked in pen testing or offensive type roles. They didn’t have a great understanding of the extent and difficulty of securing a large enterprise. When delivering their results they lacked empathy and instead opted for a “haha we owned you again” attitude at their presentations.

The blue team, driven with determination to stop the red team winning again at any cost hid aces up their sleeve. They knew the identity and location of the red team and had full visibility into their regular work actions. This led to writing specific technical detections for indicators in the red team’s malware framework that weren’t generalized against the tactics used, and searching for names and known TTP’s associated with the red team in their detection alerts.

They were then clued in when a new op was starting and caught the red team in the early stages. When the red team realized this was what was happening they got defensive, preferring now to not openly share their frameworks and code, keep new techniques hidden, creating an adversarial relationship that diminished a lot of value that could be gained from openly sharing between the teams.

Tips for preventing adversarial relationships between Red + Blue

Hire for empathy

• The people you hire should understand how difficult it is to secure an enterprise.

You don’t need the most elite hacker who is known to be a brilliant jerk. They will hurt your mission more than helping it.

Take it seriously

• People work tirelessly for years attempting to secure an organization. A security breach is not a joke.
• Be available and helpful

Make yourself available for lots of meetings post-operation. Answer all questions asked with transparency and provide any resources requested. Be prepared with timelines of actions and explanations of what was done

Schedule meetings with all the teams that will need one including:

• The engineering team for the product/system targeted
• The incident response team and the detection team etc.

Gift, thanks, and respect

• Give thanks to the blue team and highlight their wins in your communications. They worked hard during the response part of your operation and deserve a big call out for that.

IT managers have thanked me for helping give them the support to implement new security programs and changes. People even come up to me and say: “I’m scared, but when does the red team get to target us?”

If you learn how to communicate in an empathetic, humble, and lovable manner, you can tap into the cultural change and momentum that you would usually only get from a real breach.

You can keep your sharp-toothed red team that can break through a company’s defenses, and have people feel appreciative about it at the end of the day.