Skill Diversity on Incident Response Teams
So earlier this week I reposted a post about developing good interview questions for incident response. I intentionally tried to stay away from highly technical questions simply because I feel that hands-on practical interviews are better for these anyway. Hiring for an incident response team can be very difficult. Individuals need to be highly technical while still having a strategic mindset. There are few people who are able to perform both tasks. In some industries, such as DoD contracting, incident responders may have a moderately high certification requirements due to DoD 8570.01-m. We all know that highly technical people tend to avoid some certifications because they are perceived as being proof of one’s ability to memorize test questions rather then actually knowing the material.
There is another important aspect of hiring which could get lost in the chaos of trying to find good candidates. Incident response teams need a diverse set of skills to the degree that you cannot expect all members to be subject matter experts on all aspects of incident response team. When hiring outside of an entry-level position, it is important to identify what skill gaps exist inside the team and hire to fill that gap. Simply hiring another incident responder does not suffice. At a minimum, I think the following skills should be represented in every Incident Response team.
The following is a list of skills which I believe are a must in every incident response team. This does not mean every individual should hold all these skills, but all these skills should be represented on the team.
Technical Skills
Triage:
Being able to quickly triage an event is a must. When interviewing for positions at any level, some amount of technical questions should involve triaging. All members of the Incident Response Team should be able to triage an event quickly.
Host Forensics:
Incident response is essentially Digital Forensics, but performed at break-neck speeds. Whether the event was triggered by a network signature or a host signature, if the traffic was successful we will ultimately be performing host forensics to determine if the server or workstation was compromised, whether the web application was popped. Having individuals with a strong background in the host platforms used in the enterprise is necessary for successful incident response operations.
If your organization has a dedicated digital forensics team, members of the incident response team must know how to submit requests to that team. Digital forensics professionals will want to know specifically what the incident response team is looking for. When their tools are used to process images from hosts, they can write a list of keywords to automate the search. Incident responders should detail those keywords when submitting the request for analysis.
Network Forensics:
Most often it is the network signatures which begin the incident response process. On the host, mature properly configured anti-virus does a typical good job of preventing known malware from executing. Those which slip through the cracks are almost always detected by network activity. It is a must that the team have several members, if not all members, strong on network forensics. This goes well beyond the port/protocol trivia (which I think is silly anyway). This is a holistic approach to network forensics involving packet analysis.
IDS/IPS Configuration:
Organizations will typically have a dedicated team for managing the network sensors. Performing tasks such as updates and applying new signatures. Incident responders must know how to interface with the network sensor team. Through the process of incident response, new network artifacts may be discovered not covered by vendor signatures. Individuals on the team need to know how to write and submit signatures for the appropriate appliances. If you are a shop that runs Snort, some members of the team must know how to write Snort signatures.
Network Infrastructure:
Beyond the IPS/IDS, there are firewalls and routers with access control lists interacting with packets as they traverse the network. Members of the team must be capable of understanding where network appliances are being used, how they should be used, and how to utilize them when investigating or responding to an incident. Those of us who have spent time in the Department of Defense will know that they utilize robust access control lists on all of their boundary devices. Understanding how traffic flows through all devices on the network is necessary for investigating incidents.
Systems Administration:
This one can sometimes be overlooked, but if you are an Active Directory shop, it is absolutely a must that there be members with a background in Active Directory administration on the team. Understanding how accounts are created and best practices for the management of these accounts can help with resolving issues with identifying the source of internal AD problems.
Also, incident responders may have to manage a laboratory environment. Individuals should know how to administer Windows, Mac, Linux environments as necessary.
Malware Analysis:
Once you get beyond the initial triage of a potential malware infection, indicators must be developed from the sample to scope other systems which may have impacted by the malicious code. Malware will have host indicators in the form of registry changes, file system changes, and hash values which can be searched for on workstations. Most malware will also have network indicators related to its command and control functionality which can be searched for through the network appliances. Taking time to develop these can significantly improve an incident response team’s intelligence.
This does not mean every malware sample needs to be fully reversed. There is heavy reliance on the 80/20 rule here as you can quickly go down the rabbit hole. Develop actionable indicators as quickly as possible and deploy signatures based on those indicators.
Programming:
At a minimum, some individuals should have a good understanding of how the Operating System and hardware interact. Programming is a great way to gain that understanding. Some individuals should have a strong scripting background to help automate incident response tasks. If your organization lacks a true incident response tool, such as Google Rapid Response or FireEye HX, you will have to rely heavily on the scripting abilities of some members.
SIEM Administration/Operation:
The SIEM is the world which the incident response team revolves. Typically there will be a tier 1 group who is responsible for monitoring the SIEM. Individuals at all levels of the team must know how the SIEM works, what logs it receives, how to write correlation rules for the SIEM, and the processes and procedures around which the incident response team uses the SIEM.
Identity and Access Management:
I already mentioned the need to understand the directory services utilized by the enterprise. Some members need to understand the principles behind identity and access management.
Soft Skills
Executive Reporting:
Reporting is just part of the job. Not all members are required to have the skill, but at least a quarter of them should. Some members may be dedicated to formal executive reporting. Having strong writing and speaking skills is necessary for those members.
Calm and Assertive:
Rarely do incident response teams have full reign of a network. They must interact with different business groups to direct response actions against individual assets. Members of the incident response team must be able to direct those actions when those responsible for performing the actions are not direct reports. Having a calm and assertive demeanor will help is driving response actions on an enterprise network.
Juggling multiple events/incidents at once will be difficult. Members must be able to organize response actions.
Drive for Self-Improvement:
Every single member of the team must have this last skill or they should find another career.
-aub
Originally published at aubsec.github.io on April 25, 2016.
