How to build an internal red team?

I built a red team for a Forbes 30 company, and now I am sharing some pointers to help you build one in your organization

prsecurity
Sep 2 · 10 min read
Esther Jiaohttps://unsplash.com/photos/FKZwWLWgGyM

Introduction

The goal of this document is to share my experience and give practical advice on how to build and operate an offensive security team aka the “red team”.

By practical I mean the day-to-day activities that a group of people should perform to be considered an effective offensive security team.

To save you time, I avoid describing edge cases and instead provide the most important tips which you can expand upon based on the specifics of your organization.

There are two parts to an offensive security team: operations and infrastructure.

Operations are a set of rules that govern how the team chooses its targets, defines the scope, executes tasks and conducts operations to achieve its goals.

Infrastructure is a combined term for command and control centers, malware, collaboration tools, servers, internal documents and exfiltrated data storage.


Operations

Mission

The mission of an offensive security team is to improve an organization’s security posture by looking at its behavioral and technical business functions from an adversarial perspective.

The technical aspect is measuring and improving detection coverage. By imitating a known adversary or creating a profile for a fictional one, the offensive security team is helping the organization to understand how well its detections and responses are working, e.g. “can we see MacOS persistence”, “would we be able to catch exfiltration via AirDrop”, “is password reuse an issue”.

The behavioral aspect is understanding the issues in the ways people are interacting with the organization. Business logic, policies and procedures are the targets of these operations, e.g. “are there opportunities to hurt organization during the hiring process (like leaving applicants without supervision), “how do we work with our vendors”, “will we be able to detect a nation-state-sponsored agent gathering data from the inside”.

Scope

Given the breadth of the mission statement, the team must have a non-restrictive scope. In the best case that means an implied consent to test all functions that an organization can legally test.

The team would not be able to operate efficiently and meet its objectives if its actions are restrained in the same fashion as the rest of the organization. Offensive security is fundamentally an art, and creativity can not thrive in constrained environments.

There are some grey areas such as employees’ personal computers or cellphones, production databases, vendor owned equipment, etc. Overall, if a system is used to perform a business function, it is in scope unless explicitly prohibited by law.

Targets

The target must be a business function, and not an isolated system or a process.

A common mistake for an organization is to limit the scope to an app or server hence destroying the value of a red team exercise by turning it into a code review or penetration test. The team must be free to move laterally between all personal computers, servers and other devices for as long as they remain uncaught or operation is active.

The target selection process must be informed by the leadership, DFIR, Threat Intelligence, Detection and Monitoring and any other teams that have an insight into an organization’s business functions and adversaries’ appetite.

Especially important is the input from the threat intelligence team about the real adversaries attacking the organization and their intentions. The responsibility of the offensive security team is to incorporate this information in the design of their own operations. However, the final decision about the targets and scopes must remain in the hands of the offensive security team and its director.

The offensive security team must retain the independence of choosing its own targets, though it would be wasteful to engineer adversaries not related to the business threat profile, or attack systems already well covered by other programs.

Chain of Command

Ideally, a director of offensive security must work directly with a CISO or other executive officer who is in charge of the organization’s security and is a strong evangelist for offensive security capabilities. If the organization’s leadership doesn’t show strong support for the offensive security team, it will be a waste of resources to create one.

The offensive security team must also have complete independence in acting on its objectives.

Within the team, the roles should be divided as applicable to the team and the organization’s size.

At a very least you need to have a director, an operator and an engineer. If resources allow, you should expand the team to include more specialized professionals reporting to intermediate lead or principal level team members.

The specializations are defined by the director based on the team’s current skill gaps. Having specializations prevents confusion in responsibilities.

Operator’s role is to use his or her knowledge of offensive security techniques to conduct operations. They are the ones who study the target, find and exploit weaknesses, and act on objectives.

Engineer’s role is to provide operators with technical capabilities, such as vulnerability research and exploitation, malware writing, command and control infrastructure, hardware devices, network connectivity, etc.

Director’s role is to work with other teams to gather input on scopes and help remediate the weaknesses in the most cost-effective way possible, shield the team from the paperwork burden usually associated with operating a red team (such as writing reports), and to advocate for the team, support and guide it with his or her vision.

Rules of Engagement

By far the most important part of operating an offensive security team is setting the right mindset. Rules of engagement for red teaming can be taken straight out of Rules of Engagement for Operation Provide Relief:

“We are not at war. Treat all persons with dignity and respect.”

Despite the adversarial mindset, the offensive security team is not adversarial to the organization. Collaboration between all parts of the business is crucial to the team’s success. The situations where the defenders spend their time chasing the offensive security team should be avoided.

The escalation procedures can vary based on operation, in general, though, the leadership should know of an operation and keep it secret from those in the field, using it as an opportunity to train against an adversary. The communication channels should be open, so when a security incident is escalated, the leadership should be able to quickly verify its origin without causing delays during a real incident.

In case of a conflict, both parties must approach the resolution understanding that if a system broke, it was a part of its design. The idea of “no one’s fault” must be reinforced by the leadership, and no one should be blamed for any accidental damage. Otherwise, the offensive security work may be seen as an audit of performance or threat to employment.

Remediation

The offensive security team is indirectly responsible for the education of other teams. Each operation should conclude with an in-depth root cause analysis of the findings and an effective and realistic remediation plan.

The plan must not be just about remediating bugs or creating tickets. Sometimes the most obvious remediation may not be the most logical one, e.g. fixing a bug vs changing the process to make it easier for the developer to write secure code. The teams should carefully consider all second-order effects of the plan, and decided if a change in policy, additional training or acquisition of new tools may be required.

An offensive security director or a dedicated person should be responsible for leading and sharing the analysis efforts.

Training & Conferences

Red team skills are expensive due to the cost of acquisition. A strong red teamer is usually someone who invested a lot of their own time into research and self-education. It takes a unique set of life circumstances, intelligence, curiosity and persistence to build offensive skills to the level where they will be valuable to an organization.

An organization must understand this and reward red team with all-paid-for trainings and conference attendance. The team members must be free in giving back to the community with minimum friction from the organization. This involves speaking at conferences, open-sourcing tools and conducting and attending training seminars.

The side effect of proper conference presence and networking is recruitment of the best people in the industry (which is a major challenge otherwise).

Performance Metrics

Unfortunately, there are no reliable quantitative metrics you can use to calculate the team performance because of the creative nature of the work.

A commonly used metrics such as a total of bugs found are inherently wrong and unrelated to the offensive security team’s mission.

That said, looking qualitatively at the results should give a good indicator of the team’s performance. You may compare the number of operations completed, the complexity and criticality of the targets, achieved results and meaningful change delivered.

Operation’s Starting Point

The starting point will vary between operations. Some internal teams usually assume they have access to one of the internal computers, while others prefer to breach the perimeter every time. There is a benefit to both approaches, where starting anew each time keeps your perimeter in checks and starting from inside saves time and money, allowing for more attention to internal parts of your organization.

Operation Logs

The team must keep a detailed journal of events during operations, including dates, times, actions taken, commands executed, where they were executed from, etc. This will help detection teams to correlate events they see on sensors with the actions of offensive security team.

It’s important to work with your organization’s detection team, because sometime the events red team generates could be hard to find due to the volume of data coming from the sensors.

Time & Intake

It’s a good idea to plan out your engagements for a year forward. How many engagements and how long they are going to last is a function of the complexity of engagement, the size of business and the size of the team. A limit of two-three month is a reasonable timeline for an operation. No red team engagement should be less than three weeks total.


Infrastructure

Security Review

One of the challenges in running a red team is realistically imitating an adversary without getting too separated from the corporate networks.

At some point, the team’s network will have direct access to organization’s valuable servers and personal computers, and must be secured.

Exfiltrated data must be encrypted in transfer and stored in a safe place. Screenshots, operations notes, discovered secrets and other products of a red team engagement must also be properly secured.

The infrastructure could be built from the mix of cloud and on-premises capabilities, for example the main command and control can be hosted inside the organization’s data center, while the traffic proxies could be deployed in the cloud. The team should have its infrastructure designs or infrastructure-as-a-code reviewed by appropriate security teams.

The team must be trusted to deploy their own infrastructure, allowing for flexibility and independence to adjust for any specific operation or adversary profile, and they should be provided with resources to do so securely.

The malware code can be also reviewed for security vulnerabilities by the appsec teams, however, there needs to be an agreement to avoid fingerprinting of the beacons and its communication protocols during the review.

Another important aspect is the procurement of dedicated operators’ laptops that are not managed by the organization. Using those will help to keep secrecy and allow for re-imaging should the hard-drives must be wiped.

Budget

Budgeting can be divided into two areas: projectable annual budget and operational expense. Headcount, equipment, training and R&D are a part of annual team’s budget. Operating expenses are the expenses incurred for each individual operation. These can include procurement of specialized tools, cloud infrastructure, source codes, etc. needed to achieve the goals of the operation.

The approval process for operational expenses needs to me simple and efficient, so the team won’t be stuck waiting weeks to get required tools.

Collaboration Tools

Depending on how much privacy and realism you plan your offensive security team to have, you may want to separate its collaboration tools from the rest of the organization.

While total secrecy is not a major concern in collaborative environments, certain operations may be more sensitive and require to use tools not accessible by the rest of the organization, e.g. in mergers and acquisitions, insider threat simulations, operations against the office of CEO or board of directors.

For the day-to-day operations, a company’s enterprise collaboration software can be used, like cloud storage, chats or similar tools for note keeping, reporting and other business activities.

Fuzzing / Password Cracking Rigs

The team is likely to need password cracking capabilities. They may choose to use a cloud provider’s GPU instances or build a password cracking rig. If the team is also doing vulnerability research, they will need dedicated fuzzing servers.

Exfiltrated Information Storage

The information gathered during operations could be roughly divided into two sets: vulnerabilities and commercial secrets.

Vulnerabilities are means for the team to advance in the operation, such as the information about misconfigurations, bugs, passwords, etc. The vulnerabilities could be safely kept in the notes.

Commercial secrets is the data that can cause harm to the organization if stolen. The exfiltrated data must only be transferred via encrypted channels and stored in secure locations. Some organizations may choose to exfiltrate random data of equal size to prove the point without endangering the organization’s secrets or to not exfiltrate anything at all.


Conclusion

Offensive security is an important part of an organization’s effort to secure its proprietary information and its customers. While the offensive security teams are not very complex, they require leadership support and understanding to be executed effectively.

I hope, I was able to provide another perspective on building and operating offensive security teams. If you have any questions, feel free to DM me on Twitter or Keybase.

prsecurity

Written by

offensive security engineer

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade