Bottom-up security testing — security in all levels

Published in

OUSPG

6 min readApr 6, 2021

Security testing is often seen as a top-down activity, which starts with threat and risk assessment and proceeds towards the details of the system. However, security testing can also be done by the bottom-up approach.

TL;DR

Security testing can be done starting from the component level, continuing to the integration level, and finishing at the system level. This way it may be easier to find individual flaws than in top-down testing, which starts with system threat and risk assessment. However, the big-picture may be missing since the testing cannot be focused on the most critical components as identified by a top-down analysis. On the other hand, the defense in depth -principle calls for hardening of all components.

Security testing levels

As “normal” testing, security testing can be divided into levels: component testing, integration testing, and system testing. Component testing looks at individual components and establishes that the building blocks of the system are solid. Integration testing verifies that the components are properly integrated together. Finally, the system testing checks that the resulting system meets the end-user requirements. Figure 1 illustrates the different architectural levels and the appropriate testing activity for each level.

**Figure 1: Levels of (security) testing**

Any non-trivial system is made up of several components. Security testing each of the components separately is probably easier than trying to eat the whole cake at once. Bottom-up security testing can be applied also in a situation where the system is not ready, the risk assessment is not performed, or the full picture of the system is missing. With bottom-up security testing, one may be able to start producing tangible results faster but not waiting for the complete risk assessment to finish.

The drawback of starting from components is missing the relative importance of the components security-wise. The top-down risk-based approach should provide you the security exposure of the components. A weakness in a component may not be a vulnerability in the whole system. However, fixing weaknesses provides you defense in depth and has also other positive implications (see my blog post Turning tables with attackers; from fixing vulnerabilities into fixing weaknesses). If the number of components is high and available security testing resources are low, then it may be proper to use the top-down approach to concentrate only on the most critical components.

Component security testing

Component security testing looks at individual components of the system without considering the other components. Components can be individual programs, hosts in a network, etc, depending on the scope of the analysis. The components may be developed in-house or they may be external 3rd party components.

Security testing or analysis of a single component does not necessarily mean that the components must be run separately, but that the focus is on one component at a time. Depending on the case, it may be possible to assess the component separately, looking at the source code, compiling a debugger version of the component, etc. However, it may be too difficult to run the component meaningful way without it being part of the system.

The security analysis activities at the component level are heavily dependent on the type of the component and the system, but you should probably check at least the following aspects:

Cryptography is based on public and secure algorithms and it is correctly implemented, preferably by trusted library code.
Storing of sensitive information is appropriate. Passwords are not stored in plaintext but secured using robust standard methods. There are no hardcoded credentials, like API keys and passwords, all secrets can be changed.
Erroneous and unusual situations are tested. Input-parsing routines and APIs are fuzzed.
Secure coding practices are used, as applicable for the used programming languages, frameworks, operating system, and application domain.
Code reviews are performed, manually and/or by tools.

The security testing of 3rd party components may or may not have already been performed. Many components have a track record of past known vulnerabilities. You should take a look at the version history of the component and check the list of fixed vulnerabilities. Having known, but fixed, vulnerabilities is not necessarily a bad thing. It may simply indicate that the component has received a lot of security analysis and is under active maintenance. However, a large set of vulnerabilities should make you wonder how many are still to be discovered. A complete absence of security-related information may indicate that the component is rarely used or poorly maintained, and its security posture is unknown.

Integration security testing

Integration security testing looks at the connection between the components and the interaction of the components. There are many established secure-design principles (see my blog post Security requirements with principles) which should be visible at this level:

Minimize attack surface: Only the minimal set of components are exposed to the potential attackers.
Build defense in depth: Compromise of a component should not mean the compromise of all its dependent components. As far as feasible, components should not trust the other. Furthermore, the communication of the components should be protected, components should not place trust on shared media such as networks, open inter-process communication buses or permissive file systems.
Separate duties: Different tasks should be divided into different components. Especially when the tasks have different security goals and responsibilities.
Minimum privilege: A component should run with the minimal level of privileges required for its task.
Don’t trust external input: The components exposed through external interfaces should have robust input sanitation in place.
Fail securely: Failing of a component should not compromise its assets or other components.

System security testing

System security testing looks at the system as a whole. The risk and threat analysis identify a set of threat scenarios for the system and the security requirements counter the threats. The system security testing should test that the security requirements are met and that the identified threats are mitigated. Further, the security tester should look for the signs of other vulnerabilities. At this level the operating environment of the system must be included in the analysis, for example, if the system is intended to run on an isolated network, then the operating environment must enforce this.

Some of the higher level secure design principles are best visible at the system level:

Establish secure defaults: The system is secure with its default settings.
Do security updates: The system is updated once security flaws are exposed.
Minimize confidential data: The amount of critical information, such as personally identifiable information, is minimized and it is purged from the system after it is not necessary to hold it.
Keep it simple: The architecture of the system should be easy to understand and thus reasoning about the achieved level of security should be doable.

Security of software development

Security analysis of a system should also look at the process and tools which have been used to produce the system. A system can already be compromised in the production phase if the development environment has not been secure. As any system in production needs updates, the capability of the vendor to create new, and secure, versions of the system is very much part of the system’s overall security posture.

Some aspects of software development that have an impact on the overall security are:

Version control: The code of the system is securely stored, changes to it are tracked, and can be afterward audited to ensure the legitimacy of the changes.
Vulnerability management: There are channels to receive vulnerability information, a process for reacting to it, and a process to update the software and all its deployments.
Supply chain management: Almost all systems rely on externally developed 3rd-party components and services for their functionality. These components must be managed, the list of components should be available, the component vulnerability situation must be monitored, and components must be updated as required.
Automated testing. There exists a set of automated tests and checks performed for each new version.
Trustworthy installation and updates: The installation and update packages of the system must be protected from attacks

Acknowledgements

This work is done in the project SECREDAS (Product Security for Cross Domain Reliable Dependable Automated Systems) funded by ECSEL-JU (Electronic Component Systems for European Leadership Joint Undertaking) of the European Union’s Horizon 2020 research and innovation programme under grant agreement nr. 783119, and by Business Finland.

Special thanks to Marko for the original idea and comments!