What I’ve Learned in Over a Decade of “Red Teaming”

Historically, most of my posts have been technical in nature (and can be found on the MDSec blog — normal service will resume shortly). This time round though I’m going to have a stab at something less technically focussed and share some of my musings around “red teaming”, as well as my experiences in getting in to it (and infosec in general) and building out a red team service offering at MDSec.

You’re probably reading this and thinking, “great, exactly what the Internet needs is another debate over what red teaming is” and I totally agree (you can stop reading now :)); in general infosec has a bit of a problem around nomenclature.

What brought about this post was reading the following story by Florian Roth which I personally felt showed a fundamental misunderstanding of what a real red team operation was and what it was trying to achieve. The TLDR that I took away from the story was his view of red teaming was there should be more breadth and depth and red teamers should “dumb down” their techniques to closer simulate the TTPs of “known threat groups”:

The Problems With Today's Red Teaming

So what is a red team?

Emulation, simulation, operation, red, purple, white, black and gold; it can all be a little confusing so it’s no wonder we have a nomenclature problem.

The subject of what constitutes a red team seems to be particularly nuanced, depending on who you ask, you will get a variety of answers. Certainly the impression I get from talking to people based in the US and listening to the many security related podcasts/talks, is that many consider red teaming to be much more focussed around physical assessment. In my experience, this is very infrequently the case.

Rather than trying to explain exactly what I consider a red team to be, I’m going to suggest that this is already a somewhat answered question, or at least in Europe. Sometime back in 2014, Bank of England released a framework called CBEST which outlined an approach for conducting “cyber resilience” simulations. This was shortly followed by similar frameworks, TIBER-NL in 2016 and the pan-european TIBER-EU in 2018; these were backed by De Nederlandsche Bank and European Central Bank respectively.

These frameworks provide a threat intelligence (TI) driven approach, for performing red team operations. The implementation guides are public domain and referenced above, but to summarise the premise is that TI is used to define a number of scenarios that are most likely to be leveraged by an adversary when attacking an organisation. The focus of the engagements is always directed against a set of objectives, typically orientated around the critical functions that underpin the institutions continuity. During the engagements, there is always only a very limited subset of the organisation that is aware that it is on-going, and one of the key concepts is to remain undetected as long as possible. There is also always the opportunity to “dechain” and assume breach by introducing a foothold if the red team is unable to reliably achieve or consistently maintain initial access.

The scenarios should be tailored to the organisation and backed by intelligence; that is there should be something actionable highlighting that the specific adversary or adversaries uses these tactics to attack its targets. At a high-level, the scenarios will almost always include tactics like spear phishing, compromise through perimeter/cloud infrastructure, insider threats or physical access. They may also go as specific to outline the specific people, technology and processes each scenario should focus on or include common pre-texts or preferred victim profiles used by the most likely adversaries. The execution of the scenarios then typically follows the traditional kill-chain like approach through reconnaissance, delivery, exploitation, lateral movement and action on objectives.

As previously mentioned, the operational objectives are typically focussed on the critical functions of the organisation; some of the things we’ve been asked to achieve in the past include gaining access to payment systems such as SWIFT or Faster Payments, compromising ATM infrastructure, PCI zones, POS networks or television broadcast infrastructure and in many cases demonstrating impact, for example by modifying an in transit payment or showing access to cardholder data. It is not typically focussed around the business support technology or privilege; for example (as mentioned by Florian Roth) we don’t normally care about obtaining domain admin, unless it’s a route we absolutely have to walk to achieve the overall objective of compromising the function. Indeed, in many cases walking that path might increase your chances of detection.

The benefit of the frameworks are they provide a well structured approach to executing the engagement, outlining the expected deliverables and steps in the process; for example the recommended approach to a TIBER-EU test is:

Red team test steps taken from TIBER-EU framework guide

Although it’s required in the framework driven exercises, I personally don’t necessarily agree that the threat intelligence phase is always an essential component, and without getting in to a debate on the value of threat intelligence, in many cases I feel an experienced operator can loosely define the most appropriate scenarios and tactics by consuming publicly available intelligence such as through MITRE ATT&CK or the many publicly available breach reports. That said, if the provider gets it right and the organisation is able to supplement this through insight in to their own visibility of the attacks they see day to day, it can be a useful asset.

The purpose of the exercise is two-fold, firstly it provides the organisation with an opportunity to gauge their readiness to an organised and planned out cyber-attack. The output of the operation will typically demonstrate one or more attack paths to achieving the agreed objectives and highlighting what failures occurred along the way. Secondly, it provides the organisation with the opportunity to exercise their detection, prevention and response capabilities. This part is slightly more contentious, if the objective is to also remain undetected then you may not always provide full value here as the two concepts are contradictory. Typically, what we do for many engagements is to bring our “A game” up to the point we’ve completed all of our objectives, then if we’re still undetected and if agreeable, we weaken our tradecraft to see how far we need to go to be detected and when we are, we let the response playbooks play out. On completion of the operation we almost always try and work with the relevant stakeholders from the blue team to walkthrough a re-enactment of each step of the operation, comparing what telemetry they got to what they could have and where possible helping them improve detection rules. The goal here is knowledge transfer and to hopefully put the organisation in a better position to increase their maturity than they were beforehand.

The exercise is not intended to provide breadth and depth, I personally believe that this can be addressed much better through Purple Team assessments. During these assessments we would run through as many TTPs at each step of the kill chain, often with alignment to ATT&CK and work with the blue team to see which TTPs they spot and if they don’t, how can their resources be better tuned to do so based on either the telemetry they have or could get.

Do you need a red team?

I’ve seen many of the self-proclaimed red team thought leaders making statements like “Only companies with mature blue teams need a red team”. I understand this perspective and it’s partly accurate, but I personally think the question is a little more complex. I’ve performed red teams for many different types of companies with varying maturity, in some cases they have not even had a dedicated blue team. Yes, in most of these cases it was like kicking a puppy – however the company has still had value out of it because demonstrating the art of the possible will almost always achieve board level impact; one client called us the hammer to finally nail home the issues that had been disregarded for so long. In another case, a CISO called me a week after the project to thank me that it had opened so many eyes he’d been able to get a sit down with the CEO for the first time and it had caused the purse strings to be loosened to get him the much needed investment to build a detection and prevention capability. So personally, I don’t think it’s as black and white as, “you must have had X pentests” or “you must have a mature blue team”, there can be value for most organisations in knowing what impact a determined and equipped adversary might have. However, I do concede that the most value will certainly be obtained from the more mature organisations and personally I find the challenge of working against and with a blue team at the top of their game much more rewarding than slicing through a company’s defences like a knife through butter.

How do you procure a red team?

In the interests of full disclosure, before I dive in to this aspect it’s only fair to acknowledge that I have skin in the game as I work for @MDSecLabs and one of our core services is red team assessments.

However, I felt this to be a fairly important topic to discuss on the basis that I’ve heard so many first hand horror stories (from client’s past experiences) of “different” approaches to red teams. These have included things like sending consultants onsite and irresponsibly throwing USBs around or firing responder, metasploit and Nessus hail marys, as well broadcasting the pentest vendors name all over the network because they named their laptop <name>-<vendor>. The latter gave the blue team a good chuckle by all accounts. A proper red team engagement is not cheap, and I’d argue in the cases I just mentioned what they actually got was a pentest at best. Unfortunately, because the term is so poorly defined and every shop is offering their variant of a “red team”, this has also really started to devalue it.

The aforementioned CBEST and TIBER frameworks actually also released procurement guides ( CBEST and TIBER). They outline the core concepts for evaluation criteria for your red team provider including amongst other things, reputation, expertise in the domain, certification and accreditation, staff competence and the key one for me, R&D capability.

As a fast evolving and innovative subject area, providers at the top of their game will be backed by a competent R&D capability, with custom tools, novel tradecraft and innovation. If I was personally procuring a vendor for a red team, asking them to demonstrate the quality of their R&D contributions (and of course the ability to use these researchers in the project) would form the foundation of my decision making. If they were unable to show any research, or their highlight was a guide to installing bloodhound, then I might think twice about what value I’m getting from them.

As mentioned, there’s a large variety in quality in this space, however in my opinion there are a number of providers that not only have my respect but I believe “get it” in terms of what a red team is. This is evident in their contributions to community and public research efforts. While I’m sure there are also many more, the ones that have immediately spring to mind in no particular order are:

• MDSec (I have an obvious bias here but hopefully it’s justified :)),
• F-Secure Consulting (formerly MWR),
• Outflank,
• SpecterOps,
• Silent Break Security.

It might sound unusual to go to the lengths of calling out specific competing vendors, but it’s also good to recognise those that are contributing to the evolution of the trade.

How do you get in to red teaming?

A question I get asked a lot is, how do I become a red teamer? I’m sure there’s number of possible paths, but I’ll start of by talking about the one that worked for me.

I spent quite a few years dabbling in security, doing some exploit and tool development while working in sysadmin and software dev roles before taking my first step in to a professional pentest role at the start of 2006. The first 4 years or so I spent the majority of my time travelling to customer sites performing internal infrastructure pentests. These were typically blackbox domain compromise style engagements. This was a great learning ground for honing my skills, getting exposed to a wide range of technologies and environments. After some time I had pretty much perfected my approach so I started to try and bring more value by focusing on demonstrating impact; for example by gaining access to specific environments or critical systems. Oftentimes, the compromises were quite trivial, so I then started to try and push myself, for example by only living off the land or trying to operate without tripping any alerts. This was never done with a particular view to becoming a red teamer, but when I look back I think the skills I learned during this time gave me the foundation knowledge to step in to it. I would strongly recommend any budding red teamers do some time performing internal pentests and if you’re already doing that, don’t let yourself stagnate, try and push yourself, can you do the same thing without using metasploit? only from Linux? over a pivot? etc etc.

Some time around 2010 I was working at NGSSoftware and we started to receive requests for “APT Simulations” off the back of a spate of high profile breaches by the Elderwood Group. The nature of these breaches had really created a lot of noise and many clients were concerned if the same thing could happen to them. Being one of the senior infrastructure guys and having done some client side exploitation work, I threw my hat in the ring to tackle these. By current standards the first one we did was a bit of a shambles, there were no real publicly available implants at the time and no Cobalt Strike or equivalents you could just easily acquire so I spent most of the first week furiously developing a c++ implant and making sure it was safe against the popular AV engines. Most of the code was written in the early hours and it was incredibly basic, it simply retrieved commands every 10 seconds from a PHP API and executed them on the command line then posted back the response; it would not fair well in today’s defences and I would release the code but it might stir the offsec tools debate further and thinking back I was still in my 20s working at 3am so it’s doubly contentious! Fortunately for us, at the time there were also a lot of Java exploits floating around and click to play was yet to be a thing so we repurposed some recent Java exploits and got them to drop and execute my little implant. This worked pretty well and we got our shell on almost the first phish and although the implants features were basic, we were able to live off the land as well introduce tools to the environment using WebDAV.

This brings me to my second recommendation on becoming a red teamer, spend some time developing tools, build your own implants and appreciate how they work. This will really help you not only develop custom tooling when the situation requires it, but also get a better understanding of how the tools you’re using actually work under the hood.

After this is it became a semi-frequent request from clients and each time my tradecraft improved. Around 2015 following the release of the regulator driven frameworks, we witnessed a significant spike in the demand and since then it’s been almost back to back red team work.

That was my pathway in to red teaming, as I mentioned I’m sure there are many others, this is just what worked for me. In my mind, there’s two types of skills required for red teamers and some people may only possess one or the other, but both skills are in my opinion required within the team for a successful red team engagement; operational skills and development skills.

Let’s look at some topics for each of these you might want to learn, alongside some of the resources that might help you plug any gaps:

Operational:

• Active Directory: AD underpins the majority of most modern organisation’s infrastructure and understanding the opportunities for lateral movement and escalation within AD is an almost essential skill for any red teamer. Recommended resources: https://adsecurity.org/
• Tradecraft OpSec: being able to get a foothold in to an organisation is all well and good, but if your post-exploitation opsec is weak you’ll loose your access pretty quickly in most mature environments. Learning how to operate and the indicators that your tools emit will blend in with the noise. Recommended resources: https://www.youtube.com/watch?v=l8nkXCOYQC4&feature=youtu.be and https://blog.cobaltstrike.com/2017/10/25/modern-defenses-and-you/
• C2 frameworks: master your C2, understand it’s features and limitations. Having knowledge of more than one C2 will give you greater flexibility to choose the right one when the situation dictates. Not only that, knowing your C2 inside out will streamline your operations rather than having to look up or bruteforce the right command in the heat of an operation. Recommended resources: https://www.factionc2.com/ and https://github.com/cobbr/Covenant
• Infrastructure design: efficiently building and deploying your operational infrastructure is a core aspect of a red team. Your infrastructure not only needs to be able to stand up to active response, but also should protect access to your client’s data. Recommended resources: https://github.com/bluscreenofjeff/Red-Team-Infrastructure-Wiki
• Pivoting: A key skill for evading the latest and greatest endpoint detections is being able to operate over a C2 pivot. Learning how to traverse your target’s network without heavily operating on the endpoint will not only assist in defeating EDRs but also allow you to “bring your own tools” in to the environment. Recommended resources: https://artkond.com/2017/03/23/pivoting-guide/ and https://github.com/SecureAuthCorp/impacket

Development:

• Process injection: remote process injection is a powerful way of blending your operations in to an environment, whether that’s sliding your implant in to a hidden process or just masking your post-exploitation actions. However, injection techniques are also well understood by defenders, knowing the IoCs associated with each injection strategy and being able to develop your own custom loaders is a valuable skillset to possess. Recommended resources: https://ired.team/offensive-security/code-injection-process-injection, https://blog.xpnsec.com/undersanding-and-evading-get-injectedthread/ and https://www.endgame.com/blog/technical-blog/ten-process-injection-techniques-technical-survey-common-and-trending-process
• Implant development: there are lots of great open source and commercially available C2 frameworks and developing an implant that is as feature rich as Cobalt Strike, Covenant et al is no small feat. However, I would still recommend at some point taking time to develop even a basic C2 framework, doing so will help you appreciate how they work under the hood but also give you ideas on innovating and customising other frameworks. Having a custom C2 will pay dividends in the long run, in particular when detections are built to hunt the popular C2s. Recommended resources: https://posts.specterops.io/entering-a-covenant-net-command-and-control-e11038bcf462 and https://posts.specterops.io/designing-peer-to-peer-command-and-control-ad2c61740456
• Payload creation: unless you’re assuming breach, obtaining initial access is one of the most complex aspects of a red team engagement. Many of the public payload generation frameworks are signatured and will have little success against a mature, modern enterprise. If you’re able to develop your own custom payloads, you stand a much greater chance of getting that precious beacon for your team. Recommended resources: https://www.mdsec.co.uk/2019/02/macros-and-more-with-sharpshooter-v2-0/, https://www.mdsec.co.uk/2018/06/freestyling-with-sharpshooter-v1-0/ and https://www.youtube.com/watch?v=xY2DIRfqNvA

Finally, as with many other disciplines a pathway to getting the career you’re looking for can be assisted with certification. As someone in a hiring position, I recognise that certification is not the be all and end all, indeed many of the best people I’ve worked with in the past have had no formal certification. However, it can certainly do you no harm in differentiating yourself from other candidates or at least helping you get your foot in the door. The certifications in the red team space are fairly sparse, although you may want to consider the CREST CSAS, CREST CSAM or Pentester Academy Red Team Expert.