SOC2 Suck — A False Sense of Security

THE BEGINNING

SOC2 is a well-known information security compliance framework and certification. In my experience, it is the most requested by potential customers and can expedite the purchasing process. SOC2 is an audit that allows you to prove the claims you make about the security of your company are grounded in truth.

SOC2, which stands for Service Organization Control 2, is a set of auditing standards developed by the American Institute of Certified Public Accountants (AICPA) to evaluate an organization’s security, availability, processing integrity, confidentiality, and privacy controls. SOC2 compliance and certification provide a framework for assessing and managing the security risks associated with the storage, processing, and transmission of sensitive data.

The certification process involves a comprehensive audit of an organization’s security controls and requires ongoing monitoring and testing to ensure continued compliance. Achieving SOC2 certification can help organizations demonstrate their commitment to maintaining high standards of security and can provide assurance to customers that their data is being handled in a secure and responsible manner.

GreyNoise is officially SOC2 compliant after significant work from myself and the engineering team. For some insight into how we approached the framework and advice for startups, check out this blog.

The following are my unfiltered thoughts on the sometimes good, mostly bad, and always ugly parts of the SOC2 compliance framework and process.

THE GOOD

The most important part of SOC2 is the crossroads where you choose your attitude toward security and compliance. Do you want to do the minimum and pass the audit, or do you want to take the SOC2 framework and figure out what and where you can learn from it? If you choose the latter, the process will produce illuminating value rather than a waste of time — a tiny glimmer if you squint hard enough.

There is what can be considered a basic foundation for security which is important for all companies to have in place. SOC2 compliance ensures that that foundation exists. Choose the hard road, make difficult changes to process from deep self-evaluation, and benefit in the long term. Is this security?

SOC2 will require you to implement some sort of single sign-on, CI/CD, change management/approval, code/branch protection, centralized logging, MDM, and documented policy coverage. Is this security?

SOC2 forces you to explain your security model and security practices. You may discover things that make you rethink certain processes and implement improvements. It helps to see everything laid out in some 400-item questionnaire, so you can identify problems, gaps, and future projects. You will end up cleaning resources that were created years ago and forgotten about. Is this security?

This is security.

THE BAD

Imagine you are going through a SOC2 audit for your company’s product. Every control is passing, except for one — none of your data is encrypted at rest. An unwitting employee opens a malicious email attachment, and you get ransomware. Every file on every device gets encrypted and your network is compromised. Congratulations, you are now SOC2 compliant! Is this security?

For categorizing the risk of each piece of third-party software, platforms, and vendors we use, I set out to collect compliance reports from each. Collecting these reports for higher-risk vendors is a SOC2 requirement. NDAs were signed, reports were provided, and questions were asked. One of our vendors, who is also a customer, asked for our own compliance reports. It may be valuable for the output of these reports to inform the risk level you assign to a vendor. SOC2 compliance requires using likewise compliant vendors. It’s a self-perpetuating, self-fulfilling framework. Is this security?

At one point, our framework was failing due to a missing background check for an employee because the platform we use only accepts address formats in the United States. I have heard stories of “acceptable risk” exceptions to this control from employees working in countries where background checks are illegal, or the platform wasn’t able to collect education transcripts. Under SOC2, a company couldn’t hire someone like Aaron Swartz, or someone from Florida with a third-degree felony charge for scanning their high school network. Are they not good hires? Is this security?

SOC2 is a great sales tool. It will save time on vendor security questionnaires and assessments. In some cases, you may run into a customer that has a policy that requires compliance to even be considered for a sales engagement. You get to skip steps if you are certified, can prove you know how to jump through hoops and prove that you already have. This is valuable, but is it security?

My biggest issue with this compliance framework is how much information you are required to hand over to random organizations. Plenty of companies, including our auditors and our compliance automation vendors, have fairly detailed network and infrastructure diagrams of our product. Business processes, data flows, and some fairly proprietary information is exposed. Is this security?

This is not security.

THE UGLY

SOC2 is a surface scratcher when it comes to the stuff that matters for a security company. You have a SIEM or centralized logging, but is it working as expected? Are there written alerts or detection rules? Have you outsourced rule writing for a SIEM or IDS so that when it sucks — and it will — your organization doesn’t take the blame? It doesn’t matter for SOC2.

You have MDM, but the security policies haven’t been applied in weeks. An employee left months ago and hasn’t returned their laptop. A few people are exempt from certain security policies because they work with malware samples. It doesn’t matter for SOC2.

Your documented CI/CD policy is not in line with your actual CI/CD process. Maybe none of your policies reflect implemented reality. As long as both documentation and process exist, it doesn’t matter for SOC2.

SOC2 is about sales — it’s a sales tool.

To paraphrase Thomas Ptacek: Compliance is not security. Compliance is a byproduct of security engineering. Good security engineering has little to do with compliance.

Most of SOC2 is documentation — providing evidence and screenshots — no matter their relationship with the actual reality. The people reviewing these proofs are auditors, most of the time with limited or nonexistent security or product background. Auditing companies are more likely to hire someone with a degree in accounting or direct experience with the SOC2 framework than a security engineer.

We wound up encrypting literally everything at rest to match our encryption policy and to make the control go from red to green. Our data is available for free at https://viz.greynoise.io. Our data does not contain any personal identification information about customers. It’s information that is freely accessible to anyone on the internet. Does it all need to be encrypted?

Several administrative system settings on employee laptops were locked down when we implemented SOC2 policies through our MDM. The other day, I received a message in Slack asking “where do I submit a ticket to change my screensaver?”. Sure, malicious screen savers are a great way to maintain persistence, but something about this just doesn’t feel right.

Phishing tests and training came up a few times during our audit. We do not and will not perform internal phishing tests. Recent studies and data suggest these tests are largely ineffective. Employees are trained on how to identify phishing and report it, but I will never run an internal phishing test. Some customers balk when they come across this fact in risk assessments. Companies have gone so far as to tie performance metrics, bonuses, and punitive measures to the outcome of phishing tests or phishing report numbers.

I don’t believe in introducing perverse incentives into security by punishing or rewarding users for the failures or successes of the security team’s tools and training. It puts too much responsibility on the users while undermining the delicate trust forged between security, IT, and the rest of the organization. Look, it’s my gravestone, right on this hill, “here lies Reid, he knew but he didn’t KnowBe4”.

THE END

SOC2 is generally about the security of a company itself, not its products or services. It was developed by the American Institute of Certified Public Accountants.

The software space is far too varied in terms of offerings and complexity to have a single security evaluation framework that works perfectly for everyone, especially one developed by accountants. We can do better.

I can’t air grievances without suggesting improvements.

• Embed seasoned infrastructure and application security engineers to review processes live during an audit window, not just observing accountants. It may seem like I use “accountant” as a bad word — it’s not. They just need help.
• Filing taxes is different for a married couple with a million dollars in income and a single self-employed business owner — you need the whole story to do it right. A SOC2 audit of a multi-billion dollar EDR company should not be the same as one for a Series A threat intelligence startup.
• Develop separate standards for different deployment and tenancy models. I was asked how we separate our customer tenancies and data. We don’t currently have customer tenancies, and we don’t collect customer data. We provide data, that’s our product.
• Focus more on the success of the security process and tooling and less on the documentation and proofs
• An audit should include the review of the actual live process. For example, demonstrating the process for responding to an alert, rather than just providing screenshots which prove that the alert generation exists.
• An entire industry vertical with billions of dollars in total addressable market exists just to make evidence collection easier and take less time. The evidence process should be easy enough that this market ceases to exist, or cut to 10% of its current size. It’s difficult to advocate for essentially erasing money from the economy, but it will help in the longer term. Think of it like a Fed interest rate hike for the compliance automation industry. This could potentially be accomplished if these tools were marketed to and bought by audit firms, not the individual software companies undergoing the audit.
• SOC2 compliance requirements can be somewhat vague, which may lead to inconsistencies in how organizations interpret and implement them. Conversations happen between the company and auditor about “skipped” or “acceptable risk” exceptions. These are included in the final report as a “pass” for subjective reasons only known to the company and the auditor. The reasoning behind the acceptable risk designation is more interesting and valuable than the binary pass/fail result, and should be included in the report.
• Reduce over-reliance on checklists and self-assessments. SOC2 compliance involves a self-assessment process where organizations evaluate their own security controls against the SOC2 criteria. However, this approach can lead to a false sense of security, as organizations may focus solely on meeting the criteria rather than addressing real security risks.
• Broaden the limited audit scopes. SOC2 compliance only covers specific areas of an organization’s operations, which can lead to a narrow focus on specific areas of security. While it’s essential to have controls in place for areas like data management and physical security, other critical areas may not receive the same level of scrutiny. If you are 100% cloud-deployed and have a fully remote workforce, more focus should be spent on how you secure and authenticate employees and their devices rather than filling out dozens of “Not applicable” on your visitor access logs, CCTV, and physical or badge security system controls.
• Introduce testing and validation. While SOC2 compliance requires regular testing of security controls, these tests may not always be comprehensive or effective in identifying vulnerabilities or problems. Organizations should need to invest in additional security testing to identify potential risks and ensure that their security controls are effective as a result of SOC2, not just investing in the goal of getting the certification.
• Change the criteria to match the need. Organizations should be able to work with auditors to establish more specific criteria that are tailored to their particular security risks, and focus on the effectiveness of security controls rather than just meeting compliance requirements. Differing criteria from the baseline should be communicated in the final report.

Nike has a principle: “if we do the right things, we’ll make money damn near automatic.” Solve a problem or make it easier to deal with, and the money will show up. SOC2 is a revenue generating sales tool — more money and more deals will come your way as a result, and it will speed up your pipeline. However, the framework makes the mistake of answering some important questions out of order.

There are really only two questions that matter to a security company. Consider asking “how do we make more money?” first, and ask “how do we make it harder for the bad guys to win?” second.

The best solutions have the same answer to both questions. SOC2 compliance is one of many potential answers to only one: the money question. Now switch the order — the answer to the bad guy question becomes the answer to the money question.

In the general market, the success of a security company is not evaluated based on how much security they provide, or how well they secure systems, or how many undesired outcomes they prevent. They are evaluated based on how much money they make.

SOC2 is aligned with these principles. Both the framework and the security industry at large has become primarily focused on getting to customers’ dollars before a competing company instead of our common goal: beating the bad guys.

Until we reframe how we evaluate security companies, SOC2 may not be the compliance framework we need but it’s the one we deserve.