Securing the Software Supply Chain
The NSA’s Recommended Practices Guide for Developers
It would be hard to turn anywhere in today’s Cybersecurity media and leadership landscape and not see discussions about Software Supply Chain Security. While the topic is certainly far from novel or new, recent major incidents such as the compromise of software vendor SolarWinds, subsequent Cybersecurity Executive Order 14028 and the massive Open Source Software (OSS) security incident involving Log4j have put this topic at the top of every technology and security leaders mind. These issues are and will continue to impact both vendor/proprietary software as well as OSS (the former of which is overwhelmingly composed of the second, but that’s another topic). Section 4. of the EO was completed dedicated to Enhancing Software Supply Chain Security. The EO of course spawned a slew of subsequent activities and guidance, such as NIST’s Secure Software Development Framework (SSDF), NIST’s Cybersecurity Supply Chain Risk Management (C-SCRM) guidance and their specific response to Section 4 focusing on the Software Supply Chain in particular.
As mentioned, software supply chain attacks are far from a new phenomena, with sources such as the Cloud Native Computing Foundation (CNCF) having a Catalog of Supply Chain Compromises going back to 2003. There’s also the IQT Labs Software Supply Chain Compromises — A Living Dataset, which also dates to 2003. The latter represents IQT Lab’s attempt to produce a living dataset of publicly reported software supply chain compromises. But while software supply chain attacks may have nearly a 20 year track record, the pace at which they have been occurring is accelerating based on published research, such as this usenix article from Dan Geer, Bentz Tozer and John Speed Meyers titled “For Good Measure — Counting Broken Links: A Quant’s View of Software Supply Chain Security”.
So with the acceleration of software supply chain attacks and increased impact from notable incidents such as those mentioned above, we’ve seen both increased momentum among important supporting efforts such as Software Bill of Materials (SBOM), Vulnerability Exploitability eXchange (VEX), Security Levels for Software Artifacts (SLSA) and several others, all of which aim to help address aspects of securing the software supply chain. We’ve also seen an increase in critical guidance from organizations such as NIST along with industry commercial titans such as Google, Microsoft, AWS and now the National Security Agency (NSA).
In this article we will take a look at the NSA’s software supply chain security guidance, its various sections and its overarching takeaways. The NSA’s guidance is produced by a group dubbed the “Enduring Security Framework (ESF)” Software Supply Chain Working Panel. The ESF group itself is a public-private cross-sector working group spearheaded by the NSA and CISA producing security guidance to high priority threats particularly those to critical infrastructure. The announcement for the publication emphasizes the role the Developer plays in creating secure software and states this publication strives to help developers adopt government and industry recommendations on doing so. Subsequent releases from ESF will focus on the supplier and the software consumer, given the unique role each play in the broader software supply chain and its resilience.
At a high-level the document is organized into three parts:
- Part 1: Security Guidance for Software Developers
- Part 2: Software Supplier Considerations
- Part 3: Software Customer Recommendations
The guidance notes the unique role Developers, Suppliers and Customers play in the broader software supply chain ecosystem.
Within the context of a software producers perspective, they are the supplier, providing features and capabilities to customers, with those features and capabilities being driven and created by developers. This inevitably puts the suppliers and their development teams in the dichotomy of speed to market (or mission in the public sector) contrasted with the rigors of secure and resilient software, or software-enabled products.
As noted in the image, each of the 3 entities has respective security activities they can, and should, be doing. These activities span the gamut from initial secure software development, composition and architecture all the way through security acceptance testing and integrity validation on the Customers end.
Secure software begins with a Secure Software Development Lifecycle (SDLC) and the NSA guidance cites many options teams can utilize, such as NIST’s SSDF, CMU’s “Secure Software Development Lifecycle Processes”, and others such as the recently announced OpenSSF “Secure Software Development Fundamentals” courses.
The guidance stresses not just using secure software development processes but producing tangible artifacts and attestations which are used for validation, both by the software producer and consumer to have assurances related to the security and resiliency of the software. These processes and activities include best-practices such as Threat Modeling, SAST, DAST and Pen Testing but also utilizing secure release activities such as Digitally Signing, a notable example being the increased adoption of Sigstore which is a standard for signing, verifying and protecting software. The adoption and use of Sigstore is also cited in the OpenSSF’s Open Source Security Mobilization Plan as a method to deliver enhanced trust in the software supply chain.
Secure Product Criteria and Management
A core part of producing secure products is having organizational secure development policies and procedures in place along with the accompanying budget and schedule to facilitate secure product development. This of course presents potential friction where there is a myopic focus on speed to market or resource constraints, since the time, expertise and cost to implement some security best-practices will inevitably have a cost associated with them.
In a nod to some of the great DevSecOps work underway within the Department of Defense (DoD), the NSA guidance utilizes a Secure Software Development process diagram from the DoD’s DevSecOps Fundamentals Playbook. However, unlike the DoD’s example, which cited Dev, Sec and Ops as the three entities within the infinity loop, the NSA example overlays the three entities it discussed in the introduction of the guidance, which are Developers, Suppliers and Customers. Rather than organizational teams that contribute to the software development process (like in the DoD example), these three entities represent the various entities involved in producing and consumer secure and resilient software products and extend beyond an single organizational boundary, implying the software supply chain context intended.
The NSA guidance provides several examples of threats that may occur when developing and delivering software products such as malicious adversaries, the introduction of vulnerable third-party components, exploiting the build process and compromising the delivery mechanism what facilitates the transportation of the product to the customer. Software producers should have security-focused processes in place to mitigate these risks such as creating threat models, defining release criteria and documenting and publishing their security procedures and processes for releasing software. There are many other examples provided in the guidance which are worth reviewing as well.
Some of the practices recommended while far from new, are increasingly gaining in popularity. For example, Threat Modeling has been advocated for by industry leaders such as Adam Shostack, Robert Hurlbut and many others for several years. Threat Modeling is essentially thinking of what could go wrong and how to prevent it, and deeper from sources such as the Threat Modeling Manifesto or OWASP Threat Modeling projects. Threat Modeling can be done across the software development lifecycle and is recommended by NIST for Critical Software that the Federal government uses. Defining Critical Software was required by NIST from the Cyber EO and more guidance can be found on their software supply chain guidance page for Critical Software. Tools such as OWASP’s Threat Dragon exist as an OSS option for organizations looking to get going and can assist with activities such as data flow diagrams, ranking threats and suggesting mitigations.
NSA’s guidance heavily recommends Security Test Plans and Release Criteria. Test plans and requirements often include common security testing methods such as Static and Dynamic Application Security Testing SAST/DAST , Software Composition Analysis (SCA) of third-party software and even Penetration Testing among others. Several OSS and vendor tools exist in the ecosystem to conduct testing such as SAST, DAST and SCA. These tools are also increasingly integrated with Continuous Integration and Continuous Delivery (CI/CD) pipelines to enable “shift-left” security efforts, conducting security earlier in the SDLC, where some argue it is more economical and effective.
Having these tools and testing methods in place allows organizations to determine release criteria. Criteria could include things such as vulnerabilities exceeding the organizations risk tolerance, a lack of secure development practices or, with the increased calls for software transparency, particularly in the public sector, producing, correlating and validating an SBOM. While it is critical to have established release criteria to mitigate risk to downstream software consumers, this isn’t an easy feat and requires the acknowledgement that widely pervasive vulnerability scoring methodologies such as CVSS aren’t foolproof, tools generate plenty of false positions that impede velocity and many published CVE’s aren’t exploitable. For a deeper dive on concerns with CVSS, see the article “CVSS: Ubiquitous and Broken”. Shifting through the noise while driving down vulnerabilities and enabling velocity is a delicate art that requires an orchestration of effective people, processes and technologies. Security Researcher and writer Walter Haydock discusses some of the perils of well intentioned but poorly effective security release criteria.
The NSA recommends organizations have mature and codified product support and vulnerability handling policies and processes. These include having a vulnerability submission system, a Product Security Incident Response Team (PSIRT) and secure delivery of in-field software via secure protocols such as HTTPS/TLS.
Given software vulnerabilities inevitably originate from the individuals and teams writing the code, assessment and training is absolutely pivotal. This first requires defining policies that cover details such as who needs training, at what frequency and covering what topics. Training often includes topics such as secure software development, secure code reviews and software verification and vulnerability testing. It’s also important to ensure organizational security procedures and processes are treated as living documents that evolve and grow as the teams and organizations continue to glean lessons learned and after action reviews.
All of the mitigations and best-practices discussed above such as threat modeling, security test plans and release criteria are specifically mapped in the NSA guidance to the NIST SSDF we have previously discussed. Organizations selling software to public sector customers would be best advised to get very familiar with NIST SSDF, as it will soon require that their software production activities align with it.
Develop Secure Code
Moving on from the Secure Product Criteria and Management is the need to Develop Secure Code. This process begins with the earliest decisions such as the selection of a programming language and considering fundamentals of protecting digital systems and data such as fail-safe defaults, least privilege access control and often neglected, psychological acceptability.
Organizations should be wary of insider threats whether benign or intentionally malicious, both of which can modify or potentially exploit source code. This can be due to having developers who are simply poorly trained, or having engineers intentionally inserting backdoors into products. Engineers could be experiencing life circumstances that make them more susceptible to blackmail and compromise or they may add development features to make their life easier that lead to product insecurity. The NSA also warns of compromised remote development systems. With the proliferation of the remote work paradigm and dissolution of the network perimeter remote development endpoints are prime targets for malicious actors and organizations routinely struggle with endpoint hygiene, especially in Bring Your Own Device (BYOD) environments. There is also the potential for third-party development teams to be an attack vector. Other common attack vectors include orphaned accounts and credentials as users move on from teams and organizations but accounts remain active and able to be compromised. This reality warrants organizations having mature account and access control policies and processes.
While these scenarios can wreak havoc on development efforts and lead to insecure products, there are also mitigations organizations can take to mitigate the risk of insecure code. These involve utilizing authenticated source code check-in processes, conducting vulnerability scans as previously discussed and hardening the actual development environment, which may look different depending on the organizations operational conditions.
Something the NSA demonstrates that is also being emphasized by entities such as NIST in their 800–161/Software Supply Chain guidance is the creation of a Secure Repository Process Flow. This allows organizations to select external software components while implementing a workflow that conducts scanning, such as SCA to determine component make up and vulnerability concerns. This workflow can help vet external software components for developers and development teams and ensure an internal secure repository is established to make validated and verified components available for internal software development efforts.
Using tools such as SAST and DAST which were previously discussed align with security release criteria and identifying and mitigating vulnerabilities appropriately. Teams should also conduct nightly builds that include security and regression testing to ensure flaws or malicious changes are identified and addressed. As mentioned, developers may add features to make their life easier but that can lead to backdoors. This is where activities such as mapping features to requirements can alleviate some of the feature creep that can introduce risk. With the proliferation of remote development, the NSA recommends organizations harden their development environment. This means not only the build environment but the endpoints or systems involved in development activities or workflows. This is where controls such as VPN access, MFA, endpoint detection and response software and more can ensure remote development risk is mitigated to some extent.
Secure Development Practices should be utilized to generate, test and preserve source code. This includes the above mentioned activity of securing the development environment but also using secure development build configurations and using secure third-party software tool-chain and components.
Components and software will ultimately be integrated into the delivered product and this is why mature and secure code integration practices are followed. This includes testing third-party components for CVE’s listed in the NIST National Vulnerability Database (NVD), using security dependency analyzers using Zero Trust Principles when performing code integration.
As customers report defects or vulnerabilities, the development teams must be poised to respond to these incidents and reports. This includes having a public process to accept said reporting and internally having a process to review, diagnose and help resolve the issues. There is also the possibility that other teams add external development extensions which themselves could introduce vulnerabilities. Extensions such as those from solution teams or Value Added Resellers (VAR)’s. should also be developed with secure practices and SBOM’s should be made available to show the additional of any packages of components that weren’t included in the original product.
Verify Thirds Party Components
It’s no secret that development teams are increasing utilizing third-party software components into their code and products. This is often done to save time, cost and toil by using available software components whether COTS of FOSS. This is where having the Code Integration process discussed above can reduce the risk of introducing insecure third-party code. Other challenges include having third-party binaries, which then require conducting binary or SCA scanning to determine unknown files and OSS components that are included, along with any potentially associated vulnerabilities. SBOM verification against source code from the third party can help as well.
When selecting and integrating third-party components, an evaluation of any associated risk should occur through SCA along with other scanning methods such as SAST and DAST when source code is available. Development teams should also understand the provenance of code they ingest and the assurance of its delivery method. Provenance is done by tracing software back to the source and defining the path of the supply chain from the beginning, and varying levels of provenance requirements may be warranted depending on the organizations security requirements. For example, in the Supply Chain Levels for Software Artifacts (SLSA) framework, within the higher SLSA levels, provenance requirements are stricter.
NSA recommends obtaining components from Known and Trusted Suppliers. While we have a rich ecosystem of software producers and providers, this doesn’t mean they are equal when it comes to the quality of security of the software components. Software supply chain leaders such as Joshua Corman have actually been advocating for mimicking automobile manufacturing leaders and utilizing better & fewer suppliers which include higher quality components, such as in his USENIX talk “Continuous Acceleration: Why Continuous Everything Needs a Supply Chain Approach”.
OSS particularly is adding to the challenges of third-party component governance. As Sonatype’s “State of the Software Supply Chain 2021” report points out, we’ve seen a 73% Year-over-Year (YoY) increase in developer downloads of OSS components (2.2 trillion+ downloads) but also a 650% YoY increase in software supply chain attacks through methods such as Dependency Confusion, Typosquatting and Malicious Code Injection, as malicious actors also realize the efficiency of this attack vector.
When selecting and ingesting third-party components or OSS software, organizations need to understand any inherent risks and vulnerabilities associated with it. Organizations may take paths such as establishing a list of known trusted suppliers as well as vetted third-party components as previously discussed. Some considerations to be used include the quality of not just the component but also the supporting artifacts such as an SBOM, as well as the component owners historical responsiveness to vulnerability reports. Organizations should also understand that component maintenance is not to be neglected. Updates from suppliers may occur to address reported vulnerabilities for example, warranting updating the component if you’re a downstream consumer.
Harden The Build Environment
In addition to the discussion of hardening the development environment and systems, the build environment itself must be hardened. Build environments enable reproducible deliverables but when compromised serve as a highly efficient method to deliver compromised software, such as in the case with Solarwinds. The NSA guidance stressed that the build system be developed and maintained with the same level of security, integrity and resilience as that of the source code and the end product itself.
Malicious actors utilize a variety of exploits to compromise the build chain, such as infiltrating networks, scanning for vulnerabilities crafting and subsequently deploying an exploit. This exploit or malicious code can now be efficiently distributed to downstream software consumers unknowingly.
The NSA guidance recommends that not just the build pipeline but any systems coming into contact with the development and build process be locked down, protected from unauthorized network activities and utilizing secure configurations. There are also calls to implement controls such as version control, network segmentation, auditing and logging of any access and activities.
Building on these mitigations, advanced mitigations the NSA recommends include items such as Hermetic and Reproducible Builds. Hermetic builds are fully declarative, immutable and able to run without network access. This could also include hash verification. Reproducible builds validate that binary products are built from the same source, despite variable metadata. Reproducible is understood to mean identity input artifacts lead to identical bit-for-bit outputs. There are also emerging frameworks such as in-toto, which strives to protect the integrity of the software supply chain by verifying each task in the chain is carried out as planned, by authorized personnel only, and that the product is not tampered with in transit.
Several of the processes and practices defined by NSA build on not just the SLSA framework mentioned above but another emerging framework known as Software Component Verification Standard (SCVS) from OWASP. SCVS is led by none other than Steve Springett, who also is the creator of the widely popular Dependency Track project. SCVS has six control families which include not just the build environment but also Inventory, SBOM, Package Management, Component Analysis and Pedigree and Provenance. Similar to SLSA, SCVS utilizes varying levels which get more rigorous requirement and capability wise the further you go.
While software signing is seeing increased adoption and recognition as a crucial part of securing the software supply chain, with efforts such as Sigstore growing in adoption by major projects such as Kubernetes, there is also the possibility of an Exploited Signing Server. Cryptographic signatures help protect the integrity of software artifacts and provide assurances around their consumers. However, if the signing server and mechanism itself is compromised, those assurances aren’t trust worthy or useful. This allows malicious actors to legitimately sign potentially malicious artifacts and outputs which are ingested by unsuspecting software consumers. To mitigate this risk, the NSA recommends measures such as strong MFA and physical access control of signing infrastructure as well as signing on isolated network segments and hardening the actual signing infrastructure. Looking at recent incidents, like the one that impacted Twillio and many other SaaS providers demonstrates how MFA isn’t enough, but organizations should be implementing phishing resistant forms of MFA, such as Yubikeys.
Delivering Code
Last but not least is the act of software suppliers actually delivering code via distribution systems. Software developers should be utilizing binary composition analysis to verify the contents of the software their ship, along with any associated metadata such as versioning. By using SCA tooling, organizations can ensure no software of unknown provenance is included and that no sensitive data such as secrets have been included in final artifacts for delivery. Once again, there is a call for producing an SBOM to be included in the delivery package to the software consumer.
Malicious actors may try and leverage tactics to compromise software packages and updates as they transit the software distribution systems. This could be methods such as man-in-the-middle (MiTM) attacks to try and inject malicious code or vulnerabilities for later exploitation. Organizations can mitigate these risks by using secure transit mechanisms as well as using hashing and digital signatures at both the product and software component level.
Lastly, there is the potential to compromise the distribution system directly. This may include the repository, package manager and other methods to ensure compromised packages get delivered to customers. Software development teams and organizations can mitigate these threats by protecting all forms of code from unauthorized access including repositories and package managers as well as using appropriate transport layer security, such as encryption for the distribution system.
Appendices
Before wrapping up this article, it is worth noting that the NSA guidance provides a robust set of appendices which are very valuable. This includes:
- A Crosswalk between threat scenarios discussed and the NIST SSDF, mapping specific practices and mitigations to SSDF for either the Developer, Supplier or Customer.
- Dependencies: These are dependencies and artifacts that may be provided by either the supplier, third-party suppliers or customers to benefit suppliers and developers.
- SLSA — A breakdown of what SLSA is and the various Requirements, Descriptions and Levels.
- Artifacts and Checklists — Artifact Examples and Descriptions and Purposes such as Product Readiness Checklists, Design Documents and Build Logs. The guidance explains through examples the value these artifacts can provide and to whom.
- Informative Resources — The guidance provides a robust set of references from sources such as BSIMM, DoD CIO, Cyber EO, OWASP and many others. Many of these references were cited throughout the document as examples of existing guidance that the NSA leveraged and built upon.
