Incident Response: Lessons from the Trenches

Over the past seventeen years, I have successfully manage security incidents that cover a broad spectrum of threats. These range from highly capable advanced persistent threats through to opportunistic and untargeted attacks using commonly available exploit code. While all incidents are their own unique creations and the true nature of the incident only becomes clear during the course of the investigation, through those years of hands on experience, I have identified four key lessons that will make the management of incidents easier and more effective.

The First Lesson. Prevention is better than cure.

There really is no excuse for being the third, fourth or even the 100th company breached using the same MO (Modus operandi).

Groups conducting attacks, whether for financial gain or other motives, will frequently use the same methods of compromise. This is demonstrated in the recent attacks utilising SQL injection to compromise externally facing applications to expose sensitive data, and the on-going use of targeted phishing emails to gain access to corporate networks, amongst others. The use of similar methods by attackers means that organisations have an opportunity to identify attack approaches and vulnerabilities that could be applicable to them. Organisations should therefore look to use the experiences of others within their sector to enhance their own incident management procedures.

While the full details of the incident will not be publicly available, organisations can gain insight into the incidents of others through information sharing forums (for example within the UK there is the National Cyber Security Centre led Information Sharing Partnership ‘CiSP’) and employees’ individual relationships with their counterparts in other organisations. It is likely that an organisation will be able to gain sufficient information to identify the vulnerabilities exploited by attackers and key attack vectors. Using the information available, an organisation can identify potential attack scenarios and whether they are likely to be breached as a result. Organisations can then take preventative steps to remediate exposure prior to real life exploitation.

The Second Lesson. Not all incidents require a forensic approach.

Many people are conditioned to believe that any response to an incident automatically requires a forensic-led response. However, organisations need to remember that as long as they are making informed decisions then they can decide on the approach that fits best within the context of the incident or the required outcome. This could result in a more intelligence-led approach being undertaken, a forensic-led approach or a blend of both being utilised. This point is summarised well by Andy Settle (Senior Managing Consultant — Cyber Security & Threat Intelligence Solutions, IBM);

Any ‘incident’, by its very nature, requires a response which is founded upon timely and informed decision making. There are no luxuries, incidents bring with them the constraints of time, resources and limited and at times conflicting information. Although a ‘forensic led’ approach is commonly accepted for incident response, so too is ‘intelligence led’. What both of these approaches accept is the need for insight and understanding to underpin decisions throughout the stages of an incident. Rather than endeavouring to take an ‘intelligence’ or ‘forensic’ approach, better yet to take a ‘business led’ approach. One which returns the organisation back to business-as-usual in the most appropriate manner. Incident response is a business support function in time of adverse and hostile conditions, understanding that this will never be resolved with a prescriptive one-size-fits-all approach is the beginning of a mature and robust business-continuity strategy.

Forensic investigations by their nature are more detailed and methodical, as they are often focused on obtaining information and evidence that could be used within a court of law for the purpose of prosecution of a crime. Requiring suitably trained professionals and following established data handling procedures. Taking a forensic approach will thereby take longer to establish the facts and likely result in additional incident costs. Organisations should therefore take the decision at an early stage whether they wish to take the case to court or involve law enforcement. Therefore, a key question that should be asked at the beginning of any incident is:

Is there a possibility that this incident will be taken to court or involve law enforcement?

If there is any possible outcome that turns the answer to this question into a yes then you would have no choice but to use an approach that meets the evidential handling requirements of the local legal jurisdiction of the incident. Within the UK the foundations for this approach have been well documented by the Association of Chief Police Officers (ACPO) and serve to ensure that evidence handling, investigation practices and supporting activity are carried out legally.

If your organisation only needs to understand the facts around an incident and has no requirement to involve law enforcement, an intelligence-led approach may be more suitable.

Taking an intelligence-led approach broadens the tools and overall options available as part of an incident response. It will enable your organisation to gain a rapid understanding of the size and complexity of the event without the overheads of a forensic investigation. This can often result in the ability to contain or even stop the attack at an earlier stage, or even isolate compromised systems and therefore protect the wider environment.

Such an approach also enables the use of individuals with experience of incident work as well as wider security testing capabilities and attack centric knowledge. Through this broader understanding and capability, it is possible to identify and understand attack vectors rapidly. In many cases, a blended approach will often be the more suitable option. In one real life incident, a substantial fraud had been committed and law enforcement were involved from the start of the investigation. As such, a comprehensive forensic response was required to deal with the incident. As a result, this approach caused a delay with regards to understanding the initial attack vector. Without this knowledge, the organisation was not in a position to react to any possible wider exposure. Therefore, a more agile stream of work being undertaken in parallel to the forensic stream of work was called for.

In this example, it was possible to gain access to logs that would not form part of the forensic investigation. Through rapid analysis of these logs, utilising the knowledge of security testers, it was possible to identify the initial point of compromise. The analysis also identified the methods of compromise and established an overall timeline for the attack. This stream of work proved hugely valuable to the on-going investigation, enabling the organisation to take proactive steps to reduce exposure across the wider environment.

The Third lesson. The three C’s.

Co-ordination of a major cyber breach will require strong management of both technical teams and management teams. Therefore, organisations will require a robust approach to Command, Control and Communication. The key aspect of all of this will be to free up the technical troops to actually get on with the task. If they are joining incident calls every 30 minutes or even hourly, then the response work is just not going to get done.

In all of the incidents that I have managed, I always recommend that the organisation forms two incident cells, a management focused cell and a technical cell. With the technical response team separate from both of these.

The management cell are responsible for making any business decision required, setting the overall direction of the incident response and in any engagement with the media / public relations or external law enforcement. Their role is to balance what can / should be done to resolve an incident, agree priorities against wider business considerations, take any risk based decisions and authorise technical activity and budget spend. The management cell will also need to take in to account the changing nature of the incident and adjust priorities accordingly.

The technical cell is responsible for handing off tasks to the response team, to provide a buffer between the response team and the management function, allowing traction to be made.

You can then have a representative from the technical team attend the management incident calls to provide co-ordination of activity and inform and advise the management team on the technical aspects of the incident.

Lastly, I would advise having a nominated individual who is responsible within each cell for collating actions and maintaining an action log with named owners. This has two main benefits, firstly it avoids actions that are agreed verbally at 3am in the morning from being lost in the overall noise generated by an incident. Secondly, by acting as an agenda for future update calls and providing structure around those meetings.

Once an incident has been concluded, action logs provide an excellent resource for any post incident reviews.

The Fourth Lesson. Shifting sands.

All incidents are their own creations and the true nature of the incident will only become clear during the course of the investigation. As such, what you think you are facing on day one, may not be the reality later on. It is important to accept that you will have to work with limited and often constantly evolving information. However, by establishing the extent of the compromise zone at an early point, an incident response team can place a boundary around the investigation and thereby reduce potentially unnecessary effort. Therefore, cutting costs and leading to more informed decisions being taken.

Technical activity that you can expect to deploy while looking to establish the compromise zone may include:

  • Log, host and traffic analysis.
  • Security assessments of compromised environments to identify potential onward points of compromise.
  • Bespoke vulnerability research.
  • Forensic analysis of compromised assets.
  • Malware analysis.
  • Security architecture review to identify potential compromise zones.
  • Intelligence-led and incident focused analysis of compromised assets.
  • Forensic analysis of compromised assets.
In Summary. Know your enemy and know yourself.

As we have seen, often attacks methods are known in advance and knowledge of current attacks can be used to reduce the likelihood of an incident. Where an incident has occurred, challenge the assumption that a forensic approach is the only option and establishing a strong command and control approach is vital. Lastly, understanding how an attacker exploits vulnerabilities to gain unauthorised access to systems will help understand what is going on. Security testers are great at understanding those attack vectors and theorising the “what would I do” card.

So, as part of your incident response capability, making use of these experts and when suitable taking an intelligence-led approach can help in gaining a prompt understanding of the type of incident that you are facing. Thereby enabling the organisation to make informed decisions that will lead to a proportional response and ultimately in establishing the compromise zone and gaining rapid control of an incident.