Over the past years, application security has emerged as a critical field of business and IT. With software eating the world, applications are taking over the daily business. Sensitive data is stored and processed millions of times over. Governments are digitizing their infrastructure and sensitive processes like getting a receipt and buying medicine are getting digitized.
It’s no small challenge for organizations looking to deliver maximum protection for systems and data — and develop secure quality code. As a result, many organizations look into the field of application security for help, hoping to increases protection and decreases risk.
Some development teams start to embrace a security culture and hire application security development champions. Other organizations hire security consultant and perform regular workshops. Other teams have security specialists that perform regular pentests inhouse or try to harden the devices.
With all these approaches and tasks in the appsec field it is easy to get lost in the doing. It often seems easier to do things right than to identify the right things. And sometimes, for example, when a vulnerability like Log4J occurs, you have no choice but to simply act.
However, this article is looking at the ideas of the security governance competency and tries to adopt them into the field of application security. The core idea is to have a strong security governance for application development, identifying the right things to do them right. If we even make use of metrics, it will help us answer questions like ‘What should we focus our IT investments on?’
The idea to structure the development process, set up a healthy government structure and even measure the success is not new. Major Security Development Lifecycle (SDL) frameworks like BSIMM, Microsoft SDL or the OWASP SAMM all include steps to integrate governance. These frameworks help you establish a structured process for developing applications in your organization.
The BSIMM12 is a software security framework used to organize over 100 activities. The framework consists of different practices organized into the four domains Governance, Intelligence, SSDL Touchpoints and Deployment. Especially relevant here is the governance domain with its Strategy & Metrics practice. The practice encompasses planning, assigning roles and responsibilities, identifying software security goals, determining budgets, and identifying metrics and software release conditions.
The Microsoft SDL requires to have a bug bar, which is roughly similar to a vulnerability remediation plan. OWASP SAMM on the other hand requires as part of it’s Strategy & Metrics section with the lowest security maturity level to ‘Identify objectives and means of measuring effectiveness of the security program’.
Now, how can such such objectives and means of measuring effectiveness of the security program look like?
With a SDL framework as mentioned before, the maturity of the process itself could be easily tracked. An excel sheet with an honest rating once a month would be enough. However, this would only track the process implementation and not give us a usable metric for the application development itself.
In the following I will discuss a few metrics that are not only extremely insightful but should be monitored.
- test coverage
- number of crashes (fuzzing)
- number of open vulnerabilities & severity
- average time to fix vulnerability
- code quality and technical debt
- number of security related user stories
- hardening benchmarks
- number of policy violations
- part of total projects being monitored
Test coverage is a measure (in percent) of the degree to which the source code of a program is executed when a particular test suite is run. A program with high test coverage has more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage.
Number of crashes — fuzzing is an automated technique for software testing in which the program under test is repeatedly fed random data. The data can create situations in the operation of the program that cannot be achieved with other testing methods. Programs are often not designed for random input data and can crash unintentionally if the data is not plausible, thus also revealing vulnerabilities. Good fuzzers can track the program execution flow and evaluate different program paths. The crashes give an overview of the robustness of the program.
The number of open vulnerabilities and their severity is probably the most obvious and easily determined metric for application security. Automated vulnerability scans and further manual penetration testing can be used to continuously identify vulnerabilities. However, since the consequences of vulnerabilities can vary widely, it is useful to divide them into severity classes.
Another important metric (again, useful to be subdivided into different severity classes) is the average time to fix a vulnerability. It gives information about the security agility and indicate the unspoken internal view of vulnerabilities and how seriously they are taken. If you group them into severity classes you may find, that the project does not take medium vulnerabilities serious. The data can either be acquired directly from security scanners (time between first discovery of the vulnerability and the last occurrence of this vulnerability), or a vulnerability management tool.
While code quality gives insights about how clean and reusable the code is, the technical debt measures the mess left in a code base after a series of quick fixes have been made. There are static analyzers, that automatically scan and evaluate the code quality.
The number of security related user stories is very informative about the security awareness of the agile teams. In a security aware product development team, security related user stories should regularly pop up. If not, it is a good indicator of a security blind spot.
Hardening benchmarks like the CIS benchmarks, published by the Center for Internet Security (CIS), will help you identify how far away your systems are from best practice configurations.
Infrastructure as code makes it easier to define policies and track their implementation. The number of policy violations reveals if these policies are known, communicated and respected. Some tools only enforce policies and do not track violations but if possible, keep track of them and have an eye on it.
Last but not least, it is important to display the part of all projects for which monitoring has been created. Especially in large projects it will not be possible to track all projects at the beginning, so this number can be used to show the progress of the application security quantification efforts itself.
Feel free to monitor these metrics to keep track of your application security journey. Good luck and until next time.