Security Breach 102
Making your security breach public
You’ve decided to tell everyone about that security breach. So far, only the technical responders know about it. Let’s prepare for a hard couple of days.
We’ll discuss matters around employees, timing a blog post, the actual blog post, and the various other parties you’ll have to involve.
(See appendix for historic breach blog posts)
For advice on technical incident response, view Security Breach 101.
EMPLOYEE ANNOUNCEMENT
First, tell your employees what the hell is going on. Calm them down and organize communication with them well.
This is done by letting them know the nature of the incident, who is working on response, and where to direct questions or information. Most anxiety is reduced by knowing “we’re on it and taking it seriously”.
There are extreme examples of this going wrong. There was supposedly widespread employee hysteria during the Sony breach:
If the technical responders are comfortable with delivering an employee-wide message, it will let them know what is being worked and reduce panic and prevent unwanted leaks. If mail is offline or compromised, an in person all-hands meeting could work too.
TIMING
Decide on a time to release a blog post or FAQ with the technical leadership involved with the response.
You should be the first to announce the breach and control the messaging before someone with less information does it for you. Your users or customers will appreciate it. Otherwise, someone like Brian Krebs will announce the breach for you.
Do not make announcements until the technical responders have confidence that the primary adversary has no access to your systems. Technical responders will be chasing around an adversary and operating with the element of surprise (a literal ambush) against the intruder. Don’t step on that.
If this breach is already widely publicized and the intruder already knows you are responding, the element of surprise for your technical responders is much less important. You should push an announcement earlier.
Some companies have erred on the earlier side of an announcement before having all of the facts together.
You may be confident that the intruder is gone, but the technical responders will never have certainty that intruders are totally kicked out. Set your communications around this standard and include some humility with your confident statements.
Ensure you don’t have a goal of announcing “total security” after you’ve removed the attacker. It may feel as if it brings confidence back to customers, but your risk is incredibly high of a relapse.
While there are many “right” times to announce publicly, an objective indicator of doing it wrong is if your users end up having to reset passwords (or re-patch, or take any form of mitigation) more than once. Above all, do not risk having to correct an “all clear” statement.
BLOG POST
Let’s write out the blogpost. We must answer questions that users want to ask, and provide transparency to prove we’re responding seriously to the breach.
There will be lots of data that is very uncomfortable to post, but these questions will be asked or opined upon by Twitter and journalists if not directly addressed.
- What do users need to do? Reset passwords? Add multifactor? Review transactions?
- Which users are impacted? Some? All? None?
- What time window did bad actors have access to data?
- What is the specific data they accessed?
- What were the motives of the attackers?
- Who are the attackers? (otherwise known as attribution)
This point is contentious, and goes into a common “attribution” debate. The main discussion point is whether you have a standard of evidence to make a real accusation, or whether your finger pointing is a diversion from your own issues. Famous examples of breach attribution include Google pointing at China and Sony pointing at North Korea.
Do not focus too much on attribution unless you have a very clear opportunity to press charges against, or sue the bad actor. John Doe lawsuits can also be effective, but getting into them is an expensive multi-year commitment.
- What did attackers explicitly not access? If there is a sensitive area that everyone would expect to be compromised, and wasn’t, be clear with your users. Ex: Patreon pointed out specifically what wasn’t breached.
There are ways for your blog post to go above and beyond:
- Release “indicators of compromise” related to your incident. This comes from your technical responders and enables other companies to detect if the same breach occurred on their systems as well. (Of course, you can’t do this unless you’ve found them.)
- Release a post-mortem, or technical details of the breach. Some call this a Root Cause Analysis. This will expose a stronger engineering culture within your company. (Of course, don’t do this unless you can thoroughly explain what happened)
- Release features you may not have had that will improve the security of your users.
- Be transparent! It is painful to do so, but it sets you apart and proves you have nothing to hide.
Here’s the Target breach FAQ, for instance. It is incredibly thorough, answering most questions above and many others.
CUSTOMER SERVICE
After the blog post goes up, there’s no telling the amount of work that will land in your customer service organization’s inbox. Make sure they’re prepared to answer questions and have an FAQ, or you risk them making answers up based on internal word of mouth. If your business has any concept of VIP customers, you may want to reach out to them privately with special treatment before a blog post goes out.
LAWYERS
Your lawyers need to be involved. They’ll need to be prepared to explain your obligations around breach notification. Certain types of breached data will greatly influence your public response. For instance, SB-1386 is only triggered by:
Thus, lawyers need to know exactly what data was compromised to explain how much of the notification needs to follow a legal standard outside of a blog post and an email to impacted users. Victim location is a big deal and complicates your legal obligations even further.
Breach notification is a mess of intersecting and redundant laws, but it protects the consumer and you shouldn’t dodge it.
PARTNERS
You may have contracts holding you to obscure breach notifications even during vague incidents where the vendor/partner is not actually impacted. For instance, if you provide a service to a partner, they may be entitled to hear about any breach, and you may be liable for not disclosing if they discover it happened through other means.
Good lawyers would have removed this vague language from contracts. This rarely is an issue and is more relevant if you have very sensitive partnerships. Even without a contractual obligation, expect a stream of partner questions and have an FAQ prepared for confidential sharing.
PUBLIC RELATIONS
If you have PR resources or a firm on contract, they should be in touch with security+tech journalists and available to provide a comment once an announcement is out. A proactive PR team can get ahead of speculation and clickbait, ensuring that your message is out there alongside any public commentary. Breach notifications may not pick up in the press, but it’s better to assume it will.
LAW ENFORCEMENT
Contact your local FBI field office and loop them in. However, do not rely on any law enforcement agency to assist with your breach. Set your expectations that this will “check the box” and not change the situation very much.
Your team’s primary responsibility is to investigate this incident with your own manpower and resources. For crimes involving a computer, picture the FBI in a longer term investigative role. With that in mind, they will be concerned with the arsonist and not the burning building.
There are exceptions to this in specific sectors (Energy, National Security, and Financial) where various agencies are better prepared to handle an incident end-to-end, but Starting Up Security is focused more on startups that do not have those sorts of relationships.
SECURITY FIRMS
There are incidents you can handle in-house without a huge amount of external support from large security firms. (Read: Not every breach requires you to call Mandiant). Credential reuse comes to mind, or for instance, the “happiness” breach at Twitter likely didn’t need external help. Your corporate IT team doesn’t call in a firm like Mandiant for every piece of malware found on a laptop.
As a litmus test, assume any level of forensic support will demand a larger firm to be involved. If attackers have accessed multiple systems, laptops, centralized IT, etc., the need will quickly grow for manpower and specialty forensic support. Further, intrusions into source code, development systems, databases, and production infrastructure could also demand help. The knowledge base outside firms will have from handling other incidents will be useful as they frequently run into a common set of bad guys.
Security firms stand to make a lot of money on disproportionate response efforts. When involving them, be sure their involvement is proportionate to the incident‘s complexity.
DOING MORE
If the breach involved personal data that would exacerbate a fraud risk, the current trend is to provide services around credit checks and alerting.
Most breaches increase customer suspicion in passwords. Slack offered multi-factor authentication among other security features after their breach, and I believe had a great response to their incident.
Additionally, you can offer features that force resets for a group or team, display recent account activity, allow the remote destruction of sessions, or hardware tokens like Yubikey, or U2F.
Conclusion
A security breach is an emotional roller coaster. The moment you realize that you’ll have to tell the world you’ve been defeated is deeply humbling. When you’re supposedly a security professional, it will rattle your identity a bit.
My respect goes out to any in-house security pro that is expected to defend all-the-things, and then respond to all-the-incidents too.
And for those of you that are reading to prepare… You’re doing it right!
@magoo
I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.
