Stuff &Things: Non-Measurable Activities Within Infosec Programs

As cyber practitioners, we are often keen on activity. The problem is that activity doesn’t always move our cyber program if the activity isn’t measurable. This post discusses how non-measurable activities, no matter how cool or fun, are antithetical to the success of a healthy or improving cyber security program. By stuff and things, I mean activities within the cyber program that can’t be measured and described as to their value or are so non-specific in nature that the value isn’t clear.

I took to using the term, stuff & things, because I once knew of a second, or more likely, third tier cyber security software company that hands-down had the best technology solution in their market. It was amazing and had the unique problem of actually doing so many different things well that it was difficult to describe to customers. Hence, the company’s lower tier standing in the market. This company’s informal slogan that they used with customers was “we find data.” There was a great deal of truth to this but the slogan in no way conveyed how a CISO or even an engineer should think about the need or gap being filled in the cyber program. Even customers asked what that slogan meant. The slogan was about as effectively descriptive of the solution as “we find stuff” or “we do things”. In short, not very effective at all. This was the birth of the concept of “stuff & things” for me.

You may be wondering how stuff & things applies to a cyber security program. First, we need to understand human nature as it applies to cyber security teams.

CYBER TEAMS LOVE TO IDENTIFY

As information security practitioners, we love to find things. After all, “identify” or some similar version is where we love to start. We put an enormous amount of our collective energy into identification. The earlier that we can find it, the more that we can “shift left”, the better.

• Cyber risks.
• Threat hunting.
• Application security scans.
• Threat modeling.

We measure our success on what we find. Real cyber heroes find things. After all, most of the infosec social media influencers make it a point of telling our teams how we can find more stuff.

Ensuring that what is found is fixed? Not nearly the same level of interest or effort is put into fixing found stuff compared to identifying what needs to be fixed. Human nature applies to cyber security too.

There is no glory in the fixes or tracking fixes to closure. Just throw all that mundane fixing stuff into the backlog. That’s for someone else to do. No self-actualizing fun there. We’d rather do stuff & things.

But, of the two, “identifying” and “fixing”, which is the more measurable in a meaningful way? Just that small shift in focus can turn a cyber program on its head in terms of measuring and communicating progress to the executive team and Board.

NOT ALL ACTIVITIES ARE STRUCTURED TO BE MEASURABLE

A harsh reality is that every cyber program, including my own, has a set of activities that they regularly do that simply don’t tie back to pushing the cyber program forward. Many times, they seem like important things on the surface but we aren’t getting any measurable value from them.

Let me build an easy conceptual model for you. Everyone in information security is familiar with the people / process /technology model. I’ve just tweaked the outcome of process as activities. So if we focus on activity, all activities fall into one of three general categories in relation to a given cyber program:

• Activities within the cyber program that relate to other activities and contribute in a measurable way to the larger cyber program
• Activities within the cyber program that aren’t measured or don’t contribute to the larger cyber program
• Activities that are not being performed within the cyber program

The premise of this post is that the second (or middle) category is dominated by stuff & things. That category can be managed and stay flat or reduced. If left unmanaged, any given new activity that the cyber program starts has some potential to be additive to the amount of stuff and things that a program performs. At some point, the percentage of time spent in unmeasurable activities that seem to provide value can begin to exceed the amount in actual value producing measurable activities.

Yikes.

What may not be clear is that even well respected and/or well-meaning activities can meet the definition of stuff & things if they aren’t properly structured. Take patching, for instance. Obviously, patching is to be considered a value-add activity. One can normally assume that in most mid-sized or smaller organizations that patching only has so much available bandwidth per month. Let’s assume for a moment that the patching risk isn’t being measured and the patches aren’t prioritized to maximize the impact of each patch cycle. This does happen in some orgs — I’ve seen it first-hand. It would then be fair to say that the activity of patching is taking place while the outcome isn’t being measured and probably isn’t contributing much to the larger cyber program.

Fortunately, that’s not the case with patching in most organizations. That said, this is where I risk touching the third rail. Some don’t want to process the thought that their beloved cyber activity isn’t viewed as measurable or productive when the reality is that , in some cyber programs, they aren’t.

There are a number of other popular cyber activities in which organizations invest significant time that generally aren’t measured. I’m not trying to say that these activities don’t provide value; I’m saying that like the patching example above, they often aren’t being structured in a way that demonstrates their value.

ACTIVITIES PRONE TO NOT BEING MEASURABLE

I’m not trying to be a party pooper. Some things that are often treated as activities are really cool and some I used to do (or still enjoy doing) myself. But, I’m not the one that resources the cyber program nor is any CISO. That resourcing comes from outside, from the executives, from the people that need to understand what is being done in a cyber program and want answers to the question of “are we getting better?” I’m not aware of many cyber budgets that are routinely filled with enough fluff that some large percentage of effort can be expended on activities that don’t push the cyber program forward.

Where does this come into play within your program? There are three cyber areas that I most commonly see stuff & things

1. Activities with thought leadership primarily driven by tools vendors
2. Activities that are required for compliance reasons
3. Activities that are new and growing in popularity on social media

As you’ll see below, the hardest part of this is that you’ll have to understand the larger goals of your organization and how various technical activities with the cyber program can fit into the business goals.

If you don’t learn anything else from this post, please take away that there are tons of measures that are compelling to executives that aren’t expressed in financial terms. The examples below should be reasonably compelling and none will have measures available out of the box. You’ll have to think these through and build them. You hopefully also will see the high correlation to identification in the examples to activities and the same high correlation of “a path to fixing” to pushing the package forward.

Let’s dig into each area to understand why these areas have higher probability to result in stuff and things.

Activities with thought leadership driven by tools venders: Captain Obvious once confided in me that vendors that build tools want to sell you a tool. The tool itself of whatever that tool identifies/detects can become the process. And, because in many orgs, the tool has often become conflated with the process, or worse yet, the value of that process, there isn’t that critical tie-in back to measurable program level value.

Some examples of how to think differently about cyber security tools and their value:

Do you view the value of your data loss prevention solution to your cyber program as the identification of sensitive files at rest and the blocking of sensitive files when they move based on a set of business rules? You blocked X number this month, Y last month and Z the month before. Is that measuring the value? There is no real way to know if the organization is improving or a path to fixing the conditions that cause the blocks to be in place. It’s about as effective as bailing a boat with a large hole in it. You’ll always be bailing. In 5 years, you’ll still be dealing with arbitrary value of X, Y, and Z. Not measurable and not real progress for your program.

What if we changed how we think about the potential value of our data loss prevention so that we measured things that push forward the cyber program. Let’s redefine the real value of your data loss prevention solution away from the files that are being blocking and focus on the business process that allows the transfer of files that are being blocked. If there was a process within the infosec program that measured the number of DLP infractions by business unit and ended with the outcome of engaging the business unit to identify and fix the business process that allows the infractions to occur, you’d be able to measure the value of the DLP tool in terms of how many broken business processes that the cyber program has identified and helped to remediate. I’d suspect that your executive team would find that to be immensely more compelling and worthy of resourcing that how many files were blocked over a given amount of time.

A second and similar example would be with your application security program. App security is an area with thought leadership primarily driven by tool vendors. You could rethink how the app security program provides value to the organization (much like I did above with DLP) and find a set of metrics and an associated process that leads back to both identifying and fixing broken processes within the app development organization. Not easy but doable.

Activities required for compliance reasons: This one is easier to conceptually understand why we disconnect an activity from measurable cyber goals. Normal human nature is that we have more difficulty personally becoming attached to things when we are required to so. The disconnect comes when we know that we have to have to perform an activity and we assume that performing the activity makes it somehow push forward the cyber program.

Example:

If you are a longtime reader of my posts, you know that level of security culture is really owned by the business process owners. Business process owners are who drive security into the DNA of the organization. Don’t fool yourself. The level of security culture can only be impacted by you. You aren’t really able to change the security culture within your organization. The business process owners can ensure actual security culture change, though.

So why are we measuring security culture by the aspects that we control such as security awareness training? Even if people were to be consuming the mandatory security awareness content and that were not translating in measurable improvements in the cyber program, I’m not sure that we could claim that the activity of security awareness training was pushing the cyber program forward. Again, to properly measure this, you will need to understand your own business.

In my view, finding some way to measure the changes in business processes that are occurring, what business units are driving them, and which business leaders are driving them as champions would be a much more effective and measurable indicator of how security culture is changing for the better. Measuring this and communicating it broadly might also be the catalyst for enticing previously non-participating business leaders into becoming part of the change. I generate some future content on the specifics of how we do this in my organization.

Another similar example: many of us are required to have a cyber risk management program. How do you measure the success of that cyber risk program? Is the measurement similar to the DLP example in which we “measure” some never ending series of arbitrary values and make an equally arbitrary determination of improvement?

Structured properly, you can measure the value that a cyber risk management program provides for your program. Feel free to watch a full treatise on building and measuring value in cyber risk management programs .

Activities that are growing in popularity on social media: This are the most tricky of the three categories. Many new important trends start on social media. That said, social media has also generated a fair share of unsustainable fads.

My main concerns with new and often hyped activities are usually not the activity itself or whether the activity has value. My main concern is the potential opportunity cost of doing that new shiny activity that might provide some measurable value versus finishing some less sexy foundational work that will provide measurable value. A secondary but equally important concern is understanding if the prerequisites are in place in my organization to ensure that we can get the same proposed value as the organization or vendor that may be hyping the activity on social media.

A quick example comes to mind: threat hunting. Structured one way, threat hunting can be an entirely manual activity that doesn’t bring much value and like the metaphorical broken clock be right on a infrequent basis. Structured another way, threat hunting can bring measurable value to a cyber program. How do I know this? I’ve approached the threat hunting activity both ways as both a cyber leader and as a vendor. I’ve seen threat hunting rolled up beautifully into the larger goals of a cyber team, but also have seen undirected threat hunts as an activity that were complete wastes of time that took large percentages of effort away from priority projects.

That said, rather than walk through examples in this case, I’ve formalized a decision process that I personally walk through when I consider whether our cyber program needs to incorporate some new activity.

That decision process is as follows (a “yes” means proceed to the next step):

• Is this activity related to other activities in the cyber program?
• Are the outcomes of this activity measurable?
• Do the measurable outcomes connect back to key priorities in the program?
• Are all of the prerequisites in place to perform this activity?
• Should we perform this activity?
• Is there an opportunity cost to performing this activity?
• Should we perform this activity now?

As a CISO, you have to guide and shape your cyber program in two directions at the same time. You don’t just shape and communicate the program for the executive and Board but also need to guide and grow your infosec team into the best decision making about the program’s activities. The decision tree above when extended to your team can make for the conversations that they need to understand why a certain activity is greenlighted or not.

VALUE ISN’T LIMITED TO CYBER TEAMS

Services vendors also don’t seem to understand the concept of stuff & things. Also, like terrain in the military context of Sun Tzu, stuff & things can be an enemy or friend to a vendor’s business model.

Take your normal cyber security services consultant. They likely learned whatever service they provide at an organization and then left as a mid-level engineer or analyst. They are obviously skilled at what they do. But, they are highly likely to think that the value they provide is in identification — DLP blocks, SIEM alerts, cyber risks classified, threats hunted and found, etc.

The enlightened services provider will think through how to transcend the never ending identification metrics and tie back to something that the client can use to measurably tie back to cyber program objectives. You’re more likely to jump to the head of the line if you can lay out that value to the cyber decision maker(s).

Instead of increased regulation, cyber programs have to increase the amount measurable value that they receive from the various activities within the cyber program. One way would be to identify activities that aren’t producing measurable value and reduce them.

Hopefully, this blog post has given you some things to think about in terms of raising the bar for your cyber program.

The choices are yours to make.

For more insights into how cyber leaders can best enable the business and build rock solid cyber programs, please follow me on Twitter at @opinionatedsec1

You can also find more of my previous content at the “CISO & Cyber Leaders” publication on Medium: https://medium.com/ciso-cyber-leaders