Agile security testing — pentest and automate
TL;DR
Security testers are challenged by both the attackers, who actively search for vulnerabilities to exploit, and the need for test automation. Some security tests can be created automatically by tools, but mostly security testing is exploratory manual work and it is ineffective to design test cases beforehand. In this post, I explore the idea of agile security testing, where penetration testing is performed first and test cases are automated after that. Agile security testing would be made in iterations of 1) test case execution, 2) penetration testing, and 3) creation of new test cases. The iterative approach naturally leads to constantly updating tests, which addresses the problem of evolving threat landscape.
Security testing and automation
Sometimes I have been asked what is the difference between security testing and “normal” testing. Usually I emphasize that there is a lot in common, you drive to locate as many flaws as possible and provide insight into the quality of the tested product. But there is a big difference, the active attackers are looking for security vulnerabilities to be exploited. An attacker adapts and looks for different kinds of vulnerabilities when a specific approach does not work. Also, the security community constantly develops new attacks and countermeasures. This phenomenon is sometimes referred as pesticide paradox — test cases become less effective unless they are constantly updated.
The agile development process works on iterations, each of which produces at least a partially working version of the product. Fast iteration does not leave space for manual testing, so ideally all tests are automated and are run for every iteration. This is especially true with continuous integration/continuous delivery (CI/CD), where the releases are created and even deployed automatically.
The automation requirement applies to security testing. Testing of security functionality, such as login/logout, can be automated similarly to other functional testing. Some security tests can be automatically created by tools, e.g.:
- Security scans
- Fuzz testing
- Software composition analysis
Security testing also calls for exploratory security analysis of the system, the penetration testing. This analysis cannot be effectively designed and implemented as canned test cases. The tester must probe the live system to figure out the aggregate security posture of the system, which is made up of the operating system, network configuration, applications, cloud service, etc. This work includes research of the patterns, platforms, libraries, and components used to assemble the system. Looking at the stated requirements of the system is not enough, as many of the critical decisions are made in the implementation phase.
Agile security testing
In agile development, the product is developed in small iterations while getting constant feedback about the progress. Perhaps we could do the same in security testing? Below I explore this idea, which I call agile security testing.
I think it is best to start any security testing exercise with penetration testing to learn about the system and report found weaknesses or vulnerabilities. After that, we have the knowledge to develop security test cases that reproduce the key findings of the penetration testing. We can then store these test cases in the test automation repository or codify them by other means. On subsequent testing iterations, we can execute the automated test cases to check that reported flaws are fixed and there is no regression, i.e. reintroduction of already fixed issues. This cycle of 1) automated tests, 2) penetration testing, and 3) development of test cases would be then repeated continuously. This is illustrated in Figure 1.
In my mind, this approach addresses the identified problems in the automation of security testing. Test cases are not created before testing, but their creation is preceded by penetration testing of the system. This allows the tester to discover the design choices made in the implementation and tune the tests to accommodate them. In later rounds, there exists a set of automated tests, which checks that the security weaknesses identified earlier have been fixed and do not reappear. The tester can then focus on finding new weaknesses and research the new features and components added to the system. This should lead to incrementally improving security testing with automation providing persistence for the achieved gains in the security posture.
In the first iterations of security testing, it may be easier to concentrate on the security of the components which the system is made up of. Later, when understanding of the system improves, the testing could be extended to cover weaknesses and vulnerabilities in component integration and system level. This makes sense, as the weaknesses in components may be easier to pinpoint or may be already documented for the used 3rd party components. The system-level issues are more likely to be specific to the system and require threat and risk analysis to uncover. See my earlier blog post Bottom-up security testing — security in all levels for more discussion on this.
Security test cases for functional requirements can be still written beforehand and added to the test case repository, even when using the proposed agile approach. Especially if there are negative security requirements, which I discussed in the blog Reduce vulnerabilities by improving security requirements, it is probably a good idea to create automated test cases for them.
In the agile process, features are implemented incrementally and modified or sometimes removed, in later iterations. This means that the security test cases must also be modified as they become incompatible with new versions.
As I proposed in my earlier blog posts Turning tables with attackers; from fixing vulnerabilities into fixing weaknesses, it is usually more effective to test for weaknesses than to test for vulnerabilities. It may be a lot of work to research if a weakness is a vulnerability, and clever attackers have often exploited vulnerabilities that have been originally considered mere weaknesses. Further, it is likely easier to check for the presence of weakness than check for the presence of a vulnerability. For example, it is much easier to check if the port of an unessential service is open than to confirm a vulnerability in that service.
For increased efficiency, the development team should directly run the security test cases. This empowers them to debug and fix their findings without being hindered by any delays or cross-team communication overhead.
Agile triage of tools results
Security testing and analysis tools should also be used whenever possible. Static analysis, fuzzing, security scanners, software composition analysis tools, and alike, help to improve the tests effectively.
A problem with many security tools may be the large number of false positives they report. This means that they raise red flags for features of the system which are not weaknesses, but useful features. This means that the results must be triaged to find out which findings are actual weaknesses or vulnerabilities, true positives, and which are false positives.
I believe the agile security testing concept can be applied also to test tool results. In each iteration, we triage a subset of the findings and automate their triage to either give a pass or fail. In the next iteration, if the tool reports any of the same findings, we get an automatic pass or fail based on our earlier triage. A tool may have built-in support for the triage, but if not, then a separate test harness can be implemented for the tool. The goal is to run the tools, with a harness or not, for each iteration, and get increasingly accurate results from the tool as more and more of the findings are triaged.
Let’s consider an example: A port scanning tool, such as Nmap, is introduced to scan the product’s network interfaces. There appear to be 10 different open ports, some of them are required, some are unnecessary, and a few are inconclusive. A harness script is created to run the port scanner automatically. The harness gives a pass verdict when a required port is open and a fail verdict if an unnecessary port is open. Later, after the status of the inconclusive ports is resolved with the development team, the harness is updated as appropriate.
Principles of software testing
But would the agile security testing idea work? Let’s look at the Software Testing Help’s list of the seven principles of software testing [7TestingPrinciples], which I think are all relevant in security testing. I go through the principles and cross-check how the agile security testing accommodates them:
1. Testing Shows the Presence of Defects: Lack of findings is not proof of not having weaknesses or vulnerabilities in the system. ⇒ The iterative approach allows the tester to learn from the previous iterations and incrementally improve the security testing.
2. Early Testing: Exhaustive testing is not possible, one must always choose a subset of all possible tests. ⇒ Iteration allows the testing to start as soon as we have the first instance of the product ready.
3. Exhaustive Testing is Not Possible: Only a subset of all possible tests can be performed. ⇒ During the penetration testing step, the tester can make informed decisions on which tests to perform and automate.
4. Testing is Context-Dependent: Weaknesses and vulnerabilities vary from one application to another. ⇒ Tests can be tailored to the product and context.
5. Defect Clustering: Most weaknesses and vulnerabilities are likely to cluster into a subset of all components. ⇒ The tester can pay more attention to the components which on earlier iterations have looked most vulnerable.
6. Pesticide Paradox: Executing the same set of test cases several times is unlikely to find many new weaknesses or vulnerabilities. The security posture is not improved without adding new test cases. ⇒ New tests are added on subsequent iterations.
7. Absence of Error: Even when testing does not find weaknesses or vulnerabilities, there may be aspects which they are not covering. ⇒ The automated test cases are available for regression checking of the finished product. The agile security testing can be continued with new releases of the product.
So, I believe there is a good match between the challenges present in the seven principles of software testing and the agile security testing idea.
Future ideas and recap
I believe the agile security testing approach could be applied in other security analysis scenarios. For example, a penetration testing project could implement automated test cases to retest the most important findings also after the project is completed.
Security certification could also include security test cases. The test cases would be part of the certification and independently reviewed. Obviously, the product must pass all the tests. The set of test cases would allow any interested party to re-run the certification tests and observe the results. An executable certification test suite is also easy to apply for new product versions.
I hope this blog post has started you thinking about a more agile approach for security testing. Let’s make security testing agile by iterating 1) test case execution, 2) penetration testing, and 3) creation of new test cases! This approach yields constantly updating tests, which attacks the problem of evolving threat landscape.
References
[7TestingPrinciples] “7 Principles Of Software Testing: Defect Clustering And Pareto Principle”. https://www.softwaretestinghelp.com/7-principles-of-software-testing/
[Nmap] Nmap Security Scanner. https://nmap.org/
Acknowledgements
This work is done in the project SECREDAS (Product Security for Cross Domain Reliable Dependable Automated Systems) funded by ECSEL-JU (Electronic Component Systems for European Leadership Joint Undertaking) of the European Union’s Horizon 2020 research and innovation programme under grant agreement nr. 783119, and by Business Finland.
