Red Teams

Ryan McGeehan
Mar 30, 2015 · 12 min read

When you can’t find the bad guys, make some up

You've spent money on security products that escalate nothing. You have a 24/7 SOC that hardly pays attention to their tools, or knows how to use them. You have intelligence feeds but have no idea what consumes them. Logs are inaccessible, slow to query, or non-existent. Defenders have stopped hunting and lost a sense of purpose.

That means it’s time for a Red Team to come in and fuck shit up.

Why Red Teams

Observe that professional athletes compete on a schedule. A clear schedule to hone their craft to peak at any moment. Defenders have no schedule. They are called to action by any number of adversaries at any time and expected to be at a constant peak or preparedness.

Likewise, it must be satisfying after training for a boxing match to finally punch someone in the face. So while putting millions into your company’s security, you better be damn sure you’re dusting off for a battle every now and then. Otherwise there’s no emotional or tangible reward for your preparation.

Previous Work

Red Team One: “Vampire”

Let’s be very clear: to get the Red Team beyond “defense in depth”, we placed them behind defenses they’d normally have to breach. Without this it would only be a penetration test. This is cheating, and is OK.

An effective exercise must take you through a worst case scenario. Cheating helps you get there in reasonable time. It’s about the final production.

With this access, the Red Team then impersonated one of our greatest historical adversaries and threatened our security team for ~24 hours with terrifying extortion emails. This placed our team into a worst case scenario situation appearing as if the only mitigation was a complete wipe of corporate IT from servers to laptops. Defenders weren’t given a heads up about this exercise. They had to deal with it anyway.


During the response, we had to “cancel” vacation plans, calm panicked employees, and negotiate the tough decisions around production system shutdowns. Defenders were tasked with building a recovery plan which considered how much, if not all, we’d have to be rebuild. This is a significant amount of the headache you deal with when you’re really in the thick of a worst case intrusion, and has little to do with forensics or IR.

Next morning, we broke the news to the team that this was a staged (albeit real) good-guy intrusion. There was long silence, red faces, and we were doing a lot of explaining. I wasn’t sure how the team would react. A former FBI agent we hired to do malware research broke the silence with a “…Then that’s fucking awesome!” and the room lit up.

Someone then reminded the room that it was April Fools Day.

Even after explaining, several defenders came up to me saying “but we found real and unknown malware” so that “it must have been a real incident”. They were in temporary disbelief when we told them we had malware custom written for the exercise to add to the realism. People showed up late to this meeting, which caused a lot of awkward laughs as we had to break the news to every late arrival in front of everyone over and over.

That evening we had plans to take the team to a bar to decompress. They were a bit dazed for a day or so, but were absolutely pumped about the future. We now had a group of defenders who have been through the stress of a worst possible breach, together. That experience is invaluable.

Vampire murdering trophies for impactful incident responders

We rewarded the contributors of the incident response with these “Vampire” Stakes and let them know we were proud of their response. The rest of the week was cleanup and post mortem, all very calm and settled.

Deep relationships were formed between the Security and IT teams because of the shared experience. There was also a new level of empathy from IT towards Security’s goals after experiencing how bad a full intrusion could be for the company.

Evolving the Exercise

  • We made use of a browser zero day that was used to give a Red Team access to an engineer’s laptop (While we disclosed it to a vendor).
  • A former employee sat in on the Red Team, to simulate some “insider” knowledge an APT group may gain over prolonged access.
  • We asked the FBI to give us a the equivalent of a breach notification to kick off an unannounced exercise.
  • Larger production systems were added to the scope of exercises, increasing risk of a damaging mishandling.

My team was fully aware of the amount of resources we’d put into these exercises based on the previous two rounds, and knew that future exercises would involve new firepower (or we wouldn’t be doing them). They all involved strategic cheating, exploits, lateral movement, and some high value target at the end of it.

The Window

For the month leading up to the Red Team window, the team leaned towards obsession. IR Weaponization occurred. Old tools were dusted off. Comprehensive training. Re-calibrated paranoia. They wanted to beat the Red Team instead of a faceless, description-less adversary that might not even exist. Everyone wanted their contribution to be the one that caught a massive Red Team exercise.

As a result, our prevention, detection, and IR momentum increased beyond our measure at the time. We hit an even higher security peak that lasted beyond the exercise.

The Box

One Lead is All You Need

This means that after an intrusion is discovered from a single IOC, all lateral movement should be discovered if our response capability is strong.

To symbolize this, we gave responders a wooden box. The box had 10 or so wax sealed envelopes. Each envelope had an IOC that the Red Team knew about. The response team could open them whenever they needed to move things along, the goal being to open the least amount possible. This was mostly used as a time box for the game, not any measure of success. But it became a point of pride for the team to open the least amount possible.

Almost all of the IOC’s were discovered through our existing forensic and monitoring tools. This was a strong sign that we’re getting better as we probably wouldn’t have discovered them all years earlier, or at least as quickly.

Puzzle Pieces with IOC’s etched into them in envelopes to gamify the IR process

Some envelopes had a “black spot” from pirate lore, which meant that the responder had to sit out and show another team member how to respond in their place. For Red Team Four, we used puzzle pieces.

Red Team Design

“Game Master”

The Game Master should hold the Red Team accountable for their notes. Making sure they have detailed record of their intrusion is pure gold for follow up, and comparing how well the defenders discovered their badness.

This role should also be very sensitive to issues around panic, shaming, and generally how much effort and exhaustion is building up from the responders. They should have strong relationships to the organizations that manage PR, legal, any sort of law enforcement outreach, etc.

Alternate Reality Games

If you’re not familiar with ARG’s, there are many lessons to be pulled from them. It’s most important to have participated in one and understood how it can sort of wash over your normal day to day. I personally had a lot of fun with The Jejune Institute in San Francisco, but there are many online ones as well.

While participants should feel the urgency of a real incident, it shouldn’t feel so over the top that they can’t function. You want your team to stretch their incident legs with this experience, so that any future incidents run much smoother.


Don’t make physical threats part of an unannounced exercise. That goes too far. Even with an announced exercise, it could go screwy.

Vampire was high panic because it was unannounced. We didn’t allow the drills to run more than 24 hours without an “all clear — this was a drill”. They were extremely stressful this way. We would not let fully immersive drills go longer than 24 hours.

Announced exercises involving a window were able to develop panic for several reasons despite being announced. The first was that large networks and production systems were put at actual risk, to do the potential of botching the IR. Second, everyone wanted to discover what sort of new capability would be used, and be the one to catch it. So despite it being real panic — emotions and enthusiasm were positive.


Other than that — it was important to use our real tools, defenses, and systems in the exercise. None of that was pretend or table topped.


Red Team Cheating (Fourth Wall Sabotage)

So we have to cheat!

Give the Red Team important passwords. Walk them through doors, hand over design docs, or outright share a vulnerability they can exploit. Cheating effectively creates an incident to respond to, which is more important than finding actual vulnerabilities for our purposes of improving response and empathy towards security among participants.

This goes strongly against the use of a Red Team as an assessment tool. Remember — this is an exercise to improve full incident response and empathy towards security. I would argue that this more effectively improves actual security. Focusing strictly on assessments builds a policy of never-having-a-vulnerability. When measuring risk, there are no denominators to comfort you.


Have a plausible storyline with motivated adversary, their tactics, a successful intrusion, lateral movement, and exfiltration. Plan for interesting forensic artifacts in each step that a security team can discover and pivot their investigation with. Involve systems that would complicate forensics. Target production. See what happens.

Choose organizational areas with weak security and involve their leadership in planning the exercise. Making them a part of the experience will be useful. For instance, If you’re having trouble with corporate endpoint security, design an exercise around spear phishing and malware that stretches those muscles (or lack thereof). However, don’t go so far that a team is simply decimated by the exercise and is forced to observe their uselessness up close.

Time and Preparation

Security Team Knowledge

Red Teams need to be planned in absolute secrecy. The responding team cannot know a thing about what’s going on to be effective. No spoilers!

Involving senior leadership to bless the exercise pans is also a great way to involve them for remediation and post mortem functions. It will be less about getting them on the hook for helping, but getting them interested in the crazy stuff you’re about to pull.

Follow Up

Things to measure

The time lag between each response milestone are important. How quickly was a system imaged and distributed to investigators? How fast was an IOC turned around to a firewall rule? How quickly could you clear employees as non-compromised?

Understand which team members were able to run down important resources in other organizations because of their tight personal networks with other employees. Discover any bad blood between two organizations. Find the boundary spanners on your team and make sure they are appreciated for repairing any bonds.

Watch for Legal, PR, Sales, etc working without full information or approval. You can’t let a salesperson tell you “I wanted my customer to hear about the breach from me first!” when they somehow hear about something. See more about this in Security Breach 101.




They’re really fun to plan, too.


Starting Up Security

Guides for the growing security team