Almost nobody likes Penetration Tests

Increasing the Value of Penetration Tests

Nobody likes a penetration test. Well besides the penetration testers themselves, they love it! I love it! Who else has a cooler job than the person who can say that they get paid to hack into systems? Very few people.

In a penetration test, we get more insights into the actual security posture of our environment. We all want that, don’t we? Perhaps not quite:

• Programmers and administrators may be irritated by penetration tests because (1) strange people are messing around with their systems and (2) at the end of the test they will likely get a long list of things that these random people say they need to fix. To make things worse, implementing these mitigations often makes things more complicated and takes a lot of time — time that they in all likelihood never had in the first place.
• The management level may also be annoyed by penetration tests because (1) they are expensive and (2) the results of a penetration test are often simply not suitable to be consumed at a management level. What is management supposed to do with a list of 74 findings rated on the CVSS scale? The management is often not well-versed in penetration test findings' intricacies. Now they are being required to make decisions based on information that they may not fully understand.
• Product owners and various other positions within the organisation may be annoyed since penetration tests keep them from spending time on the features they urgently need to ship to customers. If they started with a certain budget and now have to expand it for a penetration test (and potential remediations) this could lead to frustration and potentially delays.

So if penetration tests are typically disliked, why still do them? Reasons may include but are not limited to:

• Compliance: various sectors require regular penetration testing for compliance reasons (regulations/legislation). We subsequently perform the test so that we get the piece of paper that says that we did it and that the system is (hopefully) secure.
• The risk department says so: theoretically a subset of compliance whereby the compliance is now driven internally. This could also include requirements from insurance providers. Further reasons may include but are not limited to the technical validation of risks and proving the security (or lack thereof) for a given system.
• Improvement: yes, there are certain instances in which a penetration test is done for the purpose of improving the security of a system (this is of course the ideal situation).

Alright, so now that we have a very high-level overview of some of the problems we face in penetration tests, let us try to summarise them:

1. The results are often so complex that only penetration testers completely understand what took place.
2. Penetration tests are expensive and often take quite a bit of time.
3. The perceived value of a penetration test is low. It is often seen as a hindrance (a hurdle that needs to be overcome before a system can go into production).

In order to attempt to solve these problems, we can derive goals for ourselves:

1. Make penetration tests understandable at all required levels
2. Reduce the costs of penetration tests
3. Create more value out of penetration tests (change the perception of mitigations)

These are the goals that we will work on in the rest of this post.

Positioning of Penetration Tests

Let us start at the beginning and remind ourselves of what a penetration test actually is. A penetration test is a technical verification of the resilience of a system against some form of attacker. We want to answer the question: is it possible for a system to be compromised by an attacker?

An important aspect that we need to consider is that the penetration test does not inherently make the system more secure. Rather, it is the processes around the penetration test that are supposed to make the system more secure.

This may seem to be a little counter-intuitive doesn’t it? The penetration test does after all tell me what I need to do to increase the security of my system? Well, yes, but it is not as simple as that:

• If the mitigations that should be implemented, are not implemented, the penetration test was in vain.
• If the mitigations are implemented but the underlying faulty processes are left unchanged, the findings will reappear and we have only secured the systems that we have tested.
• If, due to good cyber security processes (e.g. Security by Design), the systems that were tested contain no findings, the test has simply provided us with the rubber seal of approval (but has not actually done anything to improve the security of the system).
• A system may be compromised in a manner that cannot be detected in a penetration test e.g. malicious insider, social engineering, phishing, etc. The penetration test does not cover all scenarios.

It is subsequently of the utmost importance that we consider a penetration test for what it is and keep in mind what the limitations are. If we do this, it turns out that a penetration test is likely not the center of the cyber security universe (I know, this hurts fellow penetration testers).

As such, in order to use penetration tests as effectively as possible, it is important that we position them appropriately namely as a link in the larger cybersecurity process. The penetration test does not exist as its own separate process but should be intimately integrated into our existing processes.

We can in turn define the penetration test as a module that takes in certain inputs and produces certain outputs. Certain things are provided to the test and the test produces certain outputs (as seen in Figure 1) allowing the module to interact with the rest of our processes.

Figure 1: Modelling of Penetration Tests as a module within larger cyber security processes.

If we want to consider the penetration test as a module, let us define the inputs and outputs we described above:

• Input: what needs to be tested, how it needs to be tested, and for what purpose.
• Output: the results of the test (a list of findings).

Figure 2, below, highlights our inputs and outputs within our existing processes.

Figure 2: Inputs and Outputs for our Penetration Test module.

In addition, Figure 3, below, helps us to better visualise what cybersecurity processes could provide inputs to and/or receive outputs from the penetration test module.

Figure 3: Example Cyber Security Processes that could interact with penetration tests.

As a brief example, change management may require a penetration test before a change is allowed to go live. The output of a penetration test may indicate that we need to improve our awareness training procedures or improve on our secure development policies.

The key takeaway is that we need to make these connections and interactions more explicit so that we can define the benefits of tests more explicitly.

Defining inputs to penetration tests

By considering the penetration test as a module, we needed to define inputs for it. We mentioned that we want to feed it with a (1) what, (2) how, and (3) why. The question subsequently becomes, how do we define these? The keen cyber security expert may observe that we have landed on a very important topic related to penetration tests, namely, scoping.

The scoping process typically looks different from company to company and as such we will not be going into the various methods and their merits. If we remember the goals defined above, we want to make penetration tests more understandable, cheaper and increase their perceived value. How do we do this? Well by ensuring that we relate the penetration test to the critical business processes (functions).

We can use our most critical business processes (functions) to prioritise our penetration tests. Instead of testing 1000 servers, we can rather focus our penetration tests on those systems that form a part of the most critical functions within our organisation. This is useful on the scoping side as it helps us to focus and clearly define our scope (input) and prepare our findings in such a way that they are understandable at various levels (output) since they relate back to our functions.

Let us explore this concept a little further…

Scoping based on critical functions

The task of performing a penetration test may be a daunting one. As such, it is important for us to break it down into smaller sections. As described above, we can do this by means of functions:

Step 1: Functions

We need to start off by defining the most critical functions (business processes) within our organisation. The definition of the function also includes the systems that are involved in the function.

The most critical function would be where the impact on the confidentiality, integrity, or availability of the function would be the most significant (emphasis on impact).

The function is important because it provides us with a connection to our day-to-day business (the processes). A typical banking (IT) example may be card payment processing. A typical OT example, may in turn be the monitoring and controlling of the plant. If the CIA of either of these functions were to be affected, it is likely to have a significant impact on the organisation.

As such, the critical functions provide us with the critical process in the organisation that we want to assess. However, since a function is made up of entities (systems) it also provides us with the scope. Once again, the IT example may include the Payment Gateway on the Internet, the internal application server(s), and the databases/mainframes. On the OT side, we would have the operator stations and the control system itself (in different variations).

We have in turn answered the (1) what. In order to answer question (2), we need to look at the risk scenarios associated with the critical function(s).

Step 2: Risks

Each of our functions has risks associated with them. Risks answer the question of, how do I get to this worst-case impact that we defined in a previous step. Example risks for the card payment processing function may include:

• Compromise of the payment processing system that results in it being unavailable.
• Manipulation of the payment processing system resulting in the incorrect exchange of money.
• Compromise of the payment processing systems resulting in the exposure of card information.

Our OT, monitoring and controlling of the plant function may include the following examples:

• Manipulation of the control system so that the plant behaves in an unexpected manner.
• Compromise of the control system to the extent that it is rendered unavailable.

The risk scenarios, take our worst-case impacts and turn them into concrete scenarios (by adding the likelihood). The risk scenarios subsequently answer (2), the how, as they provide us with concrete scenarios that should be assessed.

Importantly, if we consider our risk to be likelihood x impact, the focus of the test will be to provide insight into the likelihood (testing for impact is covered in the Alternatives section below).

Putting it all together

Alright, so we can use functions to define the scope of our penetration test. We choose our most critical functions (the business processes that are most important to us), and the associated risks and test these scenarios. What are the advantages you may ask?

1. We do not blindly test a set of systems but have a concrete justification i.e. these systems are critical for us and as such we need to test them. In addition, the defined attacks are of particular importance due to their potential impact.
2. We do not necessarily have to test every single system in an environment (thereby potentially reducing costs). We focus on the systems that are important to us first. By testing risks, we can always ensure coverage by setting up a strategy that progressively works through the list of risks that we have. We are attempting to address goal (2) reduce the costs of penetration tests.
3. Since we are using functions, we do not test systems in isolation. In turn, we test a group of systems and determine how they could impact one another and what the likelihood of the given scenario is.
4. Our management system (specifically the risk analysis) could be the driving force behind the penetration test i.e. the scope can be defined by risks and the output of the test provides us with further insight into the risks. We are not operating a siloed approach.
5. The basis of the penetration test is a structure that everyone (especially management) can understand. Functions and risks are described at a level that everyone is able to comprehend.
6. Importantly: we get to perform a technical validation of our risks i.e. what do the likelihood and impact actually look like?

If we take a moment to visualise the process, it looks as follows:

Figure 4: Penetration test inputs

The final step is to perform the actual penetration test. The nice thing about using the risk scenarios is that on the one hand, it opens up the test type and on the other hand provides it with a focus:

• The risk scenario is described at such a level, that the exact attack vector(s) are left up to the discretion of the penetration tester, thereby not limiting their test to the working through of a checklist.
• On the other hand, it provides them with a clear goal: determine whether this scenario is possible (likelihood).

The functions, risks and entities are subsequently handed over to our penetration test as inputs where the entities can be resolved into actual systems and the final logistical issues can be taken care of.

We have taken the large problem of: you need a penetration test and have broken it down into more manageable chunks. We have progressively gone from a very abstract level (functions) down to the nitty gritty details of the actual penetration test.

Penetration Test

In this post, we will not be discussing a “Penetration Test How-To” but will rather be focusing on the interfaces we defined above. As such, if we consider our inputs, there are two approaches that we could take:

1. Keep the risk scenarios at the back of our minds and loop back to them at the end. This would involve performing a full penetration test of the in-scope systems without too much regard for the actual risk scenarios. The risk scenarios, in this case, simply provided us with the scope and the systems we need to test.
2. Perform the penetration test with the stated risk scenarios in mind. For our card processing function, this may involve testing specifically for logic attacks that may result in the incorrect exchange of money. For our OT example, we could have a look at input validation to determine what happens if I provide erroneous inputs.

Both strategies have their advantages and disadvantages. An issue with Option 2, for example, is that it may get stuck in a specific attack vector without regard for others. Option 1, on the other hand, may take a lot more time. Ultimately, it is up to the penetration testers to determine how best to answer the likelihood question associated with the provided risk scenario.

The penetration test in turn provides us with a list of findings that have been rated according to some scale (likely CVSS). What do we with these results?

Handling the outputs of a Penetration Test

If we recall problem (1) above, we noted that the results of our tests are often incredibly complex and are not particularly suited for consumption at any other level besides that of a penetration tester. Sure, the system programmer will understand banner disclosure but when we get to Cross-Site Request Forgery (CSRF), it may become a little hairy.

Now before we as penetration testers get frustrated with other people for not understanding our results let us take a minute and remind ourselves of the fact that it is in fact perfectly acceptable that not everyone can understand the results as well as we do. I can almost guarantee you that the programmer knows more about the application than what we do as the penetration tester. Likewise, the control system programmer knows their system better than we do.

However, this begs the question of whether the programmer needs to fully understand CSRF in order to mitigate it. Most modern web application frameworks handle this already and it is simply a matter of enabling it.

If we take it another step further, the manager/management also does not have to understand CSRF in order to be able to allocate resources to have it fixed. However, they have to understand it in a way that provides them with enough information to be able to take an informed decision regarding whether or to what extent it should be remediated.

As such, it is important for us to understand that we need to prepare the findings in such a way that it will move the respective parties to an appropriate level of action. This concept, in turn, fits in well with our solution to problem (1) making the results of a penetration test understandable at the required levels.

Reporting by means of risks and functions

How do we do this? If you recall, we had defined certain risks as inputs to our systems. More concretely, functions, entities, and risks.

Since we scoped our penetration test based on risks, the purpose of the test was to be able to more accurately describe the risk (likelihood x impact) associated with the scenarios. In other words, based on the penetration test results, has the likelihood of the tested risk scenario been correctly classified? Since we have actually tested the scenario (or have at least tried to as far as possible) we should be able to accurately determine the likelihood.

In the input phase, we went from the abstract level down to the detailed one. It is now time to rebuild the picture. The output from the penetration test is typically a long list of findings with CVSS scores or some other rating. The task is subsequently to chain these findings together in such a way, as to be able to provide insight into the likelihood of the risk scenario.

By means of example, if we identified a command injection vulnerability on the payment gateway that provided us with access to the backend API, we may be able to manipulate payment instructions. This may result in a high likelihood of exploitation since the attack is easy. However, if I need to compromise an administrator's account by means of a MITM attack to get access to a session token that did not have the secure flag set, the attack becomes significantly more complex and the likelihood is decreased.

As such, we can assign a likelihood to our scenario: manipulation of the payment processing system resulting in the incorrect exchange of money. Note that we could probably use this finding for various risk scenarios.

We can subsequently rebuild our picture as follows:

Figure 5: Outputs of the penetration test

The findings are chained together to link back to the risks, and the risks can be appropriately classified. The risks can, in turn, be used to define the impact on the function (should it come to this scenario) and is an ideal manner to report back to the management: dear management, based on the risk scenarios, the following worst-case scenario regarding our critical functions could occur with a high/low likelihood.

Taking Stock

Ok, so that was what quite a bit of information to take in. Let us take stock of our situation. In the beginning, we had defined three goals, namely:

1. Make penetration tests understandable at all required levels
2. Reduce the costs of penetration tests
3. Create more value out of penetration tests (change the perception of mitigations)

We have opted to use functions (business processes) as the basis of our test and the medium through which we will communicate our results. Our findings are often only understandable to us as penetration testers. By binding the penetration test to the functions and risks and by reporting on the results in terms of risks and the impact that they could have on critical functions, we have made the results more accessible to everyone who needs to be able to work with the results. Risk is a language that everyone in important positions understands. Goal 1, check.

By using the functions we are able to scope our tests in such a way that we prioritise the systems and risk scenarios that are important to us. We can define a test strategy and structure our penetration tests based on budgetary/resource/time constraints. We can test more efficiently and subsequently reduce our costs. As such, Goal 2, check.

Figure 6 provides an overview of the complete function-based process.

Figure 6: Complete function-based process

Let us take one final look back at our integration with our existing cybersecurity processes and define some example interactions more explicitly:

1. Risk Management: by starting with risk management and working back towards it, we can increase the value of the risk analysis by more accurately classifying the risks. The alternative would be to simply register all findings as risks. This may be possible but often leads to a very messy risk analysis.
2. Change Management: changes to infrastructure or products should include a risk analysis to ensure that they do not expose us to additional risk. This leads us back to the risk management described in (1) above.
3. Vulnerability Management: the output of a penetration test could be used to provide insight into outdated software that may contain vulnerabilities.
4. Awareness Training: the output of an assessment may highlight that not all attacks (risk scenarios) can be covered by technical measures. An example would be social engineering where it is imperative that users receive training. The output of the penetration test may provide us with insight into exactly which themes should be covered.

However, the important aspect is that by means of functions and risks, we are able to interface in a more effective manner.

Creating more value

Let us take a quick look at our TODO list that we defined in the beginning:

1. Make penetration tests understandable at all required levels [Completed]
2. Reduce the costs of penetration tests [Completed]
3. Create more value out of penetration tests [Open]

The final open goal is to create more value with penetration tests. The issue that we have is that even though we have made penetration tests more understandable, and reduced the cost, we still have the problem of perception i.e. great, I as the CISO understand what you did but why should I pay for the next penetration test/why should I implement the mitigations?

The second question is perhaps easy to answer: you are exposed to too much risk. But what is the actual thinking behind this?

Before we can answer this question, we first need to remind ourselves of the mandate of the management. The management is not paid to reduce risks. The management is paid to make money. Full stop. Unfortunately, zero risk != maximum profits. As such, all decisions are driven by risk vs reward. If I am exposed to too much risk, it is likely that the reward will be reduced (should the risk be realised). I can build a super cool card payments app, but if it gets hacked within 3 days, my reward is significantly lower.

So what does a penetration test do? It provides us with actual insight into the current risk landscape. The more insight I have into my risk landscape the more sure I can be of the reward associated with my product. And this is how they can sell it to the shareholders. In order to expand our product offering, improve on the product, add new features, make it faster, increase capacity, or whatever else it may be, we need a strong foundation that exposes us to the minimally acceptable risk level. The more confident I am in my risk, the more confident I can be in expanding, improving etc. my products in order to increase profit. This is how we create more value (3).

Alternatives

There are variations to the functional approach we described above. Alternatives include, but are not limited to the following:

1. If scoping is not the issue but you are having issues with reporting results to management, it is possible to simply use the output process described above to change the findings into risks that can be reported to management. Whilst a bit of reverse engineering may be required (perhaps when deriving functions) this is certainly a feasible solution.
2. We do not need a risk analysis to apply the function-based approach. In this case, during scoping we define the functions, entities, and risks.
3. If we are using the risk analysis as the input we may decide to test the implementation of requirements/mitigations instead of the scenarios. This may result in a reduction in scope because the tests are very limited. This has use cases in OT for example where it may not be possible to perform a full penetration test. This can still yield useful results, though, as it provides us with insight into the residual risk i.e. based on the current implementation status of my requirements, what is the likelihood and impact of my risk?
4. Asset-based risk assessments. The above approach can also be applied to asset-based risk assessments, however, it is a little more difficult to test attack scenarios bridging multiple systems since the risk analysis is, well asset-based. To test and document risk scenarios at an asset level may be cumbersome and unfortunately, an attacker is unlikely to compromise one system and call it a day. Nevertheless, the methodology can be applied here as well.
5. Testing for impact. In the above discussion, we placed the focus on likelihood in our risk calculation. However, there is of course also the impact (even though there are fewer mitigations that have an effect on the impact). It may be possible to test the impact depending on how the scenario is formulated. Let us consider two examples:

• Scenario 1: an attacker compromises the external perimeter. In this case, the impact may be significant as they now have access to the internal network. We could in fact test the impact i.e. what happens after they have compromised this system?
• Scenario 2: an attacker compromises the external perimeter and installs ransomware on the device. In this case, the impact would be an encrypted device — we would not learn much from actually encrypting the device.

As such, whilst it is possible to test for impact, one needs to very careful of what test will be performed as it could have undesirable consequences.

TL;DR

In the above post, we have held a very extensive discussion regarding how we could better position penetration tests. In short:

1. Penetration tests need to form part of the larger cyber security management processes (if in place). Penetration Tests should not happen in isolation but should rather be viewed as a module.
2. We can scope and report on test results by means of risks (an existing risk analysis is even better). The risks define our attack vectors and the systems that need to be tested. On the output side, risks can be used to communicate findings more effectively (and integrate back into the risk analysis if available).
3. Management is not interested in our Remote Code Execution (RCE). We need to communicate penetration test results in terms of risk so that they can make an informed decision. A firm understanding of the risk landscape enables them to improve their product more efficiently, more effectively, and with more confidence ultimately leading to increased profits.

Conclusion

If you have made it this far, well done, and thank you for staying to the end! To round up our discussion: penetration tests are an integral part of a larger cyber security strategy with the emphasis being on part. The positioning of penetration tests is integral in ensuring that they are employed as effectively as possible. It is up to us as penetration to make sure that we are able to integrate into this larger cyber security strategy and that we do not isolate ourselves.

Happy Testing!