When to purchase a ‘solution’ to your cybersecurity problem.
One thing that’s bothered me about the security industry is our propensity to immediately seek commercial solutions to our problems. Often, these commercial solutions are designed to become part of our daily security routines: they have dashboards, alerts, and other features. They have configurations or rulesets that need to be managed or tuned. Too often, a commercial ‘solution’ evolves to the point that it creates its own problems that need to be solved.
Do we question why we bought it in the first place? Do we go back and reevaluate whether we really still need it? Do we know if it is generating value? Is it effective?
In my writing and talks, I often comment that security teams with the ability to build solutions in-house seem to have more confidence in their security. I suspect part of the reason for this is that all the time that would be spent researching products, working through procurement, going through training/certifications and managing a commercial product are spent focused on solving the problem when building the solution in-house.
All of this led me to the conclusion that we should probably have a formal process for this. Several questions came to mind. How would we structure this process? How would it be prioritized? What factors would affect build vs buy decisions? How would we even know if we’re better off with a build decision versus a buy decision?
My goal here is to just share some of the ideas I came up with to spark some conversation. I’m sure I’ll miss important factors and leave out key steps in the process.
The Process
We know the security program isn’t revenue-producing, it’s all about loss and risk avoidance. Ideally, we’d want to minimize spending while maximizing the efficacy of the program, but that would ignore basic cost center economics that prevents most CISOs from doing anything to actively shrink the budget they worked so hard for. After all, next year’s needs could be completely different. We may not need 100% of our budget to be effective this year, but we might need 120% more to be at the same level of efficacy next year, right?
That’s not a rabbit hole I want to go down in this piece though. I think we can all agree that we want as much of our budget to be effective and useful as possible, so that’s the assumption I’ll work from as I describe the priorities laid out in the flowchart above.
It’s the numbered sequence (in yellow) above that is a key piece I think is missing from how we address gaps in our security programs. Most often, I’ve seen organizations go directly from research to proof-of-concept. There may be a decision process with options 3–6, but in my experience, 1 and 2 are rarely explored in a traditional IT/InfoSec environment.
There is a piece missing post-implementation also (in blue). This piece seeks to answer the question: were we successful? Even when the answer is “yes”, this process needs to remain indefinitely so that we can understand whether the process remains successful and maintains the expected level of value or performance. Again, that’s a rabbit hole best explored in a separate piece. I go into it somewhat in this primer on breach and attack simulation (BAS) I created for MISTI’s InfoSec Insider publication.
A Closer Look
We’ll go through these in the order of the process. First, the numbered process, which represents choosing the type and nature of how you’re going to solve the problem.
#1 Solve with existing resources
I’ve found that, in many cases, organizations don’t realize what their existing products can already do and rarely use them to their full potential. The result is that a product is purchased that has significant or full overlap with something the business already owned. This is often referred to as Living off the Land by both defenders and attackers.
Windows is a great example of an underutilized security tool that also has the potential to cause a lot of security problems. These days, it’s mostly because it hasn’t been hardened or because an excessive amount of convenience has been chosen over protection (which is often a valid choice, depending on business needs and priorities). This inspired me to create a workshop where I explored common and obscure techniques for hardening Windows.
#2 Build solution with existing resources
This is the one that many security teams just don’t have the resources for. There’s a broad difference in the dynamic of security teams where coding skills are common and those where this same resource is scarce. Security is, after all, a specialization that exists on top of all other IT fields, so it is generally useful to have as many of those underlying skills as possible.
Besides, coding , scripting or data analysis skills are useful on a daily basis for a security or incident response team. I haven’t personally gathered data on this, but I’m willing to bet that security teams with strong scripting or dev skills are:
- More likely to find root causes and get there faster
- Require smaller security budgets, because they’d build more and buy less (though this could be a logical fallacy on my part, since they might simply redirect the same funds towards tools to help streamline dev processes that a non-dev-savvy team wouldn’t need)
- More capable of properly prioritizing and validating vulnerabilities, making them less likely to waste other teams’ time with false positives and non-issues.
- Better rapport with non-security teams, thanks to past experience in non-security roles.
- Better grasp of business goals and ‘big picture’ context
- More likely to see through vendor BS and marketing razzle-dazzle, thanks to an understanding of the underlying components.
Applying this insight to the second option, I believe that the security industry takes advantage of the general lack of coding skills and familiarity with tools outside the security team. I’ve often seen teams purchase products that either completely overlapped with the features of products already owned by their company or that they could have easily built themselves with some open source software and/or a little elbow grease.
This brings me to an additional concern: the operational overhead typically associated with managing commercial hardware and software. Too often, I see entire security ‘teams’ that are little more than babysitters for a particular product the company owns. The WAF guy looks at WAF alerts and tunes the WAF all day. The SIEM gal looks at SIEM alerts and tunes the SIEM all day. The poor soul assigned to vulnerability management forwards scan output to various departments and is made to suffer for their chagrin.
This isn’t a team that can make ‘build or buy’ decisions. This security team simply acquires products and does what those products or vendors tell them to do. No one on this team does anything remotely approaching risk management or loss avoidance. Imagine their frustration as they see the messaging that security is the most important thing! Why are they not more respected in the organization? Why are they not taken more seriously? Misled, they might conclude that their employer doesn’t take security seriously.
The truth is that they’ve mistaken a bill of goods for a security program. This lesson is the incentive for writing this piece and is the reason a decision to build, rather than buy is #2 on this list.
#3 Buy Solution; implement and manage with existing resources
Sometimes you need a product to get the job done. I mean, we COULD make forensic copies with DD and we COULD capture packets with tcpdump… but sometimes we need FTK Imager (sorry, terrible example since it’s free) or RSA NetWitness to save us some time. Ideally, we’d do this with existing people and existing budget. Even with that best-case scenario, bringing in a new product isn’t a process for the faint-hearted, as outlined in the right-most column in the diagram above.
- You have to make the case for spending the money. This is time not spent on doing anything security-related.
- You research the available options; this can be non-trivial and you may pay an analyst firm 5–6 digits just to help with the research. This is time not spent on security work.
- You decide on your top 3–4 and devise a proof-of-concept to help with the choice between them. Most security teams aren’t skilled in setting up POCs, so it’s usually a painful process and the results aren’t terribly reliable or useful. Still, often it weeds out the true Bad Apples.
- Usually, you’re back to #1, but now making the case to both upper management and procurement. Can’t get it through a pre-approved purchase vehicle/channel? Good luck with that. Sometimes getting sponsorship from other IT groups can help with the politics.
- Implementation. In my experience, anything that takes longer than a business week to get set up and running is in danger of failing and becoming shelfware. The longer the implementation takes, the higher the chances of abandonment. In my career, I’ve seen two 18-month implementations go up in smoke literally overnight. In one case, we hired someone full time just to help with the implementation. There’s a metric I use here: time-to-value. Ask vendors what their typical time-to-value is for their other customers. Ask to speak to some of them about the experience (before buying the product, preferably!).
- Management: you mean it doesn’t just run itself after you buy it?! This is one of the most common reasons I’ve seen for security shelfware: either lacking the staff or staff with the right skills to maintain the product. Success isn’t binary with most products however, the level of effort necessary to keep a product running isn’t necessarily a level where the product produces the anticipated/required value. I have another metric that might be useful here: the value-to-labor ratio. If the vendor tells you the ideal ratio is 2:1 for their product, you need 2 full-time employees (FTE) to get full value out of the product and maintain it properly. If you only have 0.5 FTE to spare, you’ll need to skip to #5 or #6. If you’ve thrown 4 SecOps FTEs at the product, you’re either wasting resources, or using the product for a lot more than what it was designed for (which isn’t necessarily bad, you just need to adjust the ratio to suit your organization’s needs and expectations).
- Validation: Is the product doing what’s expected? Does it actually make your organization more secure? Or does the product just keep people busy? To figure this out, you first have to define what success (or ‘value’ as we called it previously) looks like. Prevents us from getting hacked is not an outcome that can be usefully measured without doing some kind of simulation, which I go into in the aforementioned post on breach and attack simulation. BAS isn’t the only way to test products and controls, however. Many controls, for example, aren’t designed to prevent external threats — they might be designed to satisfy client requirements for privacy or data protection, for example.
- Periodic reevaluation: How is the product working out 6 months after purchase? 12 months? Is it measuring up based on the metrics you put in place? Is it producing the expected value? Do you wish you had made a different choice? Are you ready to ditch it and build something in-house? Did you wish you chose a different product? Hopefully you didn’t sign a three-year contract.
#4 Acquire Service (outsource the problem)
Acquiring a service isn’t too different from #3, except that there are people-driven services involved. This is a booming market right now and managed services offerings are seemingly everywhere. Vendors are adding their own MSSP services, there are standalone MSSPs and MSSPs built out of large resellers, former telecoms and consulting giants. It has become common to see hybrid product/services models also — products designed to have customers using it and deploying it from the front end, but additional assistance with analysis or threat hunting from vendor staff on the back end.
This is a troubling trend. It implies first, that products are difficult or challenging to use without assitance or skills that don’t already exist in the customer’s organization. It also implies that all the big data, machine learning and AI we’re seeing isn’t making much of a dent in the alert fatigue problem. We still need to throw human brains and bodies at the problem, which I don’t think can scale long-term.
Still, there are many good reasons to outsource a problem:
- When you don’t have the necessary skills in-house and can’t bring them in within your budget
- You only need something temporarily
- It’s something that doesn’t add value to your security program, but compliance mandates it and it’s cheaper to outsource.
- It’s an opportunity to point the finger at someone else or shift fiscal responsibility if a breach occurs (okay, maybe they’re not all good reasons)
#5 Buy Solution; hire additional staff with existing budget
Like #3, except you’re reasonably certain you can’t manage the product with existing staff. Acquiring talent isn’t easy — I talk to many organizations that are resigned to hiring interns or bringing in candidates with backgrounds in other IT disciplines and training them up to where they need to be.
I give a talk on the future of the security practitioner and suggest that we could do with less security specialists by reassigning security responsibilities to IT subject matter experts when it makes sense.
#6 Request Additional Resources to do 1–5
Definitely a last resort, but still quite common, especially when the alert fatigue problem is far from being solved. I strongly believe that the need for a SOC is simply a symptom of this larger alert-fatigue problem.
Wrapping up
While a lot of the ideas and concepts presented here are ones that I’ve shared and talked about before, this is the first time I’ve proposed a formal process. The idea is to manage security product lifecycles leveraging a built-in feedback loop designed to minimize wasted effort and wasted budget (e.g. shelfware). I’d love to hear feedback on what I might have missed or could be simplified and consolidated. Do you think this process would be useful? Is there a better way to go about it? Has someone already built it?
