Security Breach 102
You’ve decided to tell everyone about that security breach. So far, only the technical responders know about it. Let’s prepare for a hard couple of days.
We’ll discuss matters around employees, timing a blog post, the actual blog post, and the various other parties you’ll have to involve.
(See appendix for historic breach blog posts)
For advice on technical incident response, view Security Breach 101.
Security Breach 101
I want to help you through your first security incident. It will not be easy.
First, tell your employees what the hell is going on. Calm them down and organize communication with them well.
This is done by letting them know the nature of the incident, who is working on response, and where to direct questions or information. Most anxiety is reduced by knowing “we’re on it and taking it seriously”.
There are extreme examples of this going wrong. There was supposedly widespread employee hysteria during the Sony breach:
If the technical responders are comfortable with delivering an employee-wide message, it will let them know what is being worked and reduce panic and prevent unwanted leaks. If mail is offline or compromised, an in person all-hands meeting could work too.
Decide on a time to release a blog post or FAQ with the technical leadership involved with the response.
You should be the first to announce the breach and control the messaging before someone with less information does it for you. Your users or customers will appreciate it. Otherwise, someone like Brian Krebs will announce the breach for you.
Do not make announcements until the technical responders have confidence that the primary adversary has no access to your systems. Technical responders will be chasing around an adversary and operating with the element of surprise (a literal ambush) against the intruder. Don’t step on that.
If this breach is already widely publicized and the intruder already knows you are responding, the element of surprise for your technical responders is much less important. You should push an announcement earlier.
Some companies have erred on the earlier side of an announcement before having all of the facts together.
You may be confident that the intruder is gone, but the technical responders will never have certainty that intruders are totally kicked out. Set your communications around this standard and include some humility with your confident statements.
Ensure you don’t have a goal of announcing “total security” after you’ve removed the attacker. It may feel as if it brings confidence back to customers, but your risk is incredibly high of a relapse.
While there are many “right” times to announce publicly, an objective indicator of doing it wrong is if your users end up having to reset passwords (or re-patch, or take any form of mitigation) more than once. Above all, do not risk having to correct an “all clear” statement.
Let’s write out the blogpost. We must answer questions that users want to ask, and provide transparency to prove we’re responding seriously to the breach.
There will be lots of data that is very uncomfortable to post, but these questions will be asked or opined upon by Twitter and journalists if not directly addressed.
- What do users need to do? Reset passwords? Add multifactor? Review transactions?
- Which users are impacted? Some? All? None?
- What time window did bad actors have access to data?
- What is the specific data they accessed?
- What were the motives of the attackers?
- Who are the attackers? (otherwise known as attribution)
This point is contentious, and goes into a common “attribution” debate. The main discussion point is whether you have a standard of evidence to make a real accusation, or whether your finger pointing is a diversion from your own issues. Famous examples of breach attribution include Google pointing at China and Sony pointing at North Korea.
Do not focus too much on attribution unless you have a very clear opportunity to press charges against, or sue the bad actor. John Doe lawsuits can also be effective, but getting into them is an expensive multi-year commitment.
- What did attackers explicitly not access? If there is a sensitive area that everyone would expect to be compromised, and wasn’t, be clear with your users. Ex: Patreon pointed out specifically what wasn’t breached.
There are ways for your blog post to go above and beyond:
- Release “indicators of compromise” related to your incident. This comes from your technical responders and enables other companies to detect if the same breach occurred on their systems as well. (Of course, you can’t do this unless you’ve found them.)
- Release a post-mortem, or technical details of the breach. Some call this a Root Cause Analysis. This will expose a stronger engineering culture within your company. (Of course, don’t do this unless you can thoroughly explain what happened)
- Release features you may not have had that will improve the security of your users.
- Be transparent! It is painful to do so, but it sets you apart and proves you have nothing to hide.
Here’s the Target breach FAQ, for instance. It is incredibly thorough, answering most questions above and many others.
After the blog post goes up, there’s no telling the amount of work that will land in your customer service organization’s inbox. Make sure they’re prepared to answer questions and have an FAQ, or you risk them making answers up based on internal word of mouth. If your business has any concept of VIP customers, you may want to reach out to them privately with special treatment before a blog post goes out.
Your lawyers need to be involved. They’ll need to be prepared to explain your obligations around breach notification. Certain types of breached data will greatly influence your public response. For instance, SB-1386 is only triggered by:
Thus, lawyers need to know exactly what data was compromised to explain how much of the notification needs to follow a legal standard outside of a blog post and an email to impacted users. Victim location is a big deal and complicates your legal obligations even further.
Breach notification is a mess of intersecting and redundant laws, but it protects the consumer and you shouldn’t dodge it.
You may have contracts holding you to obscure breach notifications even during vague incidents where the vendor/partner is not actually impacted. For instance, if you provide a service to a partner, they may be entitled to hear about any breach, and you may be liable for not disclosing if they discover it happened through other means.
Good lawyers would have removed this vague language from contracts. This rarely is an issue and is more relevant if you have very sensitive partnerships. Even without a contractual obligation, expect a stream of partner questions and have an FAQ prepared for confidential sharing.
If you have PR resources or a firm on contract, they should be in touch with security+tech journalists and available to provide a comment once an announcement is out. A proactive PR team can get ahead of speculation and clickbait, ensuring that your message is out there alongside any public commentary. Breach notifications may not pick up in the press, but it’s better to assume it will.
Contact your local FBI field office and loop them in. However, do not rely on any law enforcement agency to assist with your breach. Set your expectations that this will “check the box” and not change the situation very much.
Your team’s primary responsibility is to investigate this incident with your own manpower and resources. For crimes involving a computer, picture the FBI in a longer term investigative role. With that in mind, they will be concerned with the arsonist and not the burning building.
There are exceptions to this in specific sectors (Energy, National Security, and Financial) where various agencies are better prepared to handle an incident end-to-end, but Starting Up Security is focused more on startups that do not have those sorts of relationships.
There are incidents you can handle in-house without a huge amount of external support from large security firms. (Read: Not every breach requires you to call Mandiant). Credential reuse comes to mind, or for instance, the “happiness” breach at Twitter likely didn’t need external help. Your corporate IT team doesn’t call in a firm like Mandiant for every piece of malware found on a laptop.
As a litmus test, assume any level of forensic support will demand a larger firm to be involved. If attackers have accessed multiple systems, laptops, centralized IT, etc., the need will quickly grow for manpower and specialty forensic support. Further, intrusions into source code, development systems, databases, and production infrastructure could also demand help. The knowledge base outside firms will have from handling other incidents will be useful as they frequently run into a common set of bad guys.
Security firms stand to make a lot of money on disproportionate response efforts. When involving them, be sure their involvement is proportionate to the incident‘s complexity.
If the breach involved personal data that would exacerbate a fraud risk, the current trend is to provide services around credit checks and alerting.
Most breaches increase customer suspicion in passwords. Slack offered multi-factor authentication among other security features after their breach, and I believe had a great response to their incident.
Additionally, you can offer features that force resets for a group or team, display recent account activity, allow the remote destruction of sessions, or hardware tokens like Yubikey, or U2F.
A security breach is an emotional roller coaster. The moment you realize that you’ll have to tell the world you’ve been defeated is deeply humbling. When you’re supposedly a security professional, it will rattle your identity a bit.
My respect goes out to any in-house security pro that is expected to defend all-the-things, and then respond to all-the-incidents too.
And for those of you that are reading to prepare… You’re doing it right!
I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.
EXAMPLE BREACH BLOG POSTS
Linode Blog " Security Notification and Linode Manager Password Reset
January 5, 2016 1:53 pm Effective immediately, Linode Manager passwords have been expired. You will be prompted to set…
Important Security Notice from Patreon
Yesterday I learned that there was unauthorized access to a Patreon database containing user information. Our…
data breach FAQ
We truly value our relationship with you, our guests, and know this incident had a significant impact on you. We are…
Customer Notice FAQs
Here are answers to some questions you may have based on our investigation to date. Is my money at JPMorgan Chase safe…
Frequently Asked Questions on eBay Password Change
Our company recently discovered a cyberattack that comprised a small number of employee log in credentials, allowing…
August 4 Security Incident
On August 4th, 2015, we reset all of our customers’ passwords and emailed them a password reset link to create a new…
Important Hosted Chef Security Notice
Dear Customers, On Wednesday morning we became aware of a misconfiguration of an exception handler for the Hosted Chef…
Important Security Notice From MakeMusic — SmartMusic
On the afternoon of Wednesday, April 23, an attempted intrusion to MakeMusic’s computer systems was detected and as an…
Protecting People On Facebook
Facebook, like every significant internet service, is frequently targeted by those who want to disrupt or access our…
Last.fm Password Security Update — Last.fm
The world’s largest online music catalogue, powered by your scrobbles. Free listening, videos, photos, stats, charts…
Keeping our users secure | Twitter Blogs
As you may have read, there’s been a recent uptick in large-scale security attacks aimed at U.S. technology and media…
The Home Depot, Inc. — News Release
No evidence of debit PIN numbers compromised No customers liable for fraudulent charges Customers offered free ID…
Security Email | Zappos.com
Free shipping BOTH ways, 365-day return policy, 24/7 customer service. Millions of men’s shoes, women’s shoes, girl’s…
An Update on LinkedIn Member Passwords Compromised
We want to provide you with an update on this morning’s reports of stolen passwords. We can confirm that some of the…
Important Security Announcement From PagerDuty
Andrew Miklas — July 30, 2015 Our customers and community are very important to us, and to maintain the transparency…
July 2nd security incident post-mortem
On Thursday, July 2nd we discovered a serious security issue that leaked private scoped module metadata to public…
Post Mortem: Today’s Attack; Apparent Google Apps/Gmail Vulnerability; and How to Protect Yourself
This morning a hacker was able to access a customer’s account on CloudFlare and change that customer’s DNS records. The…
Post-mortem of 2013–11–15 security breach
Posted by: jik A natural outgrowth of what we do at Quantopian is that our users tend to be technically sophisticated…
Security Update | WordPress Hosting by @WPEngine
Urgent WP Engine Security Notification At WP Engine we are committed to providing robust security. We are writing today…
Bit9 and Our Customers’ Security
Earlier today we informed our customers about a potential security concern. Out of respect for our customers, we chose…
Important Customer Security Announcement
Cyber attacks are one of the unfortunate realities of doing business today. Given the profile and widespread use of…
LastPass Security Notice
Thank you for taking the time to read our posts and follow our recommended actions after the recent events. Behind-the…
Security Notice: Service-wide Password Reset — Evernote Blog
Evernote’s Operations & Security team has discovered and blocked suspicious activity on the Evernote network that…
Security Notice: Forum User Password Resets — Plex Blog
Updated July 6th, 2015: After thorough investigation by a team of forensic specialists, we’ve identified the source of…
HipChat Security Notice and Password Reset
Atlassian’s security team has discovered and blocked suspicious activity on the HipChat service that resulted in…
Recent Speculation re: LANDESK Product Security… | LANDESK User Community
LANDESK recently became aware of some unusual activity on our IT systems. With the help of a leading computer forensics…
Update on Security Incident and Additional Security Measures
What Happened On April 8, the SendGrid account of a Bitcoin-related customer was compromised and used to send phishing…
Internal security incident identified and resolved at the Wikimedia Foundation
A security incident with the Wikimedia Foundation’s Mailman mailing list system was identified and addressed today…
Important Customer Security Announcement
I want to post a quick update and apologize for the incident we have faced which may affect some of you and your…
Playing it safe: An account security update
As a precautionary measure, we’re in the process of resetting everyone’s Ting account password. We’re sorry for the…
Notice of Data Privacy Incident -
Notice of Data Privacy Incident (New York, NY, July, 24, 2015) Healthfirst is notifying approximately 5300 affected…