Is Your Mainframe Software Supply Chain Secure?
Co-authored with Vaughn Marshall
Recent software supply chain compromises of significant scale, such as Log4j & SolarWinds, have demonstrated a need to reevaluate how software is delivered and consumed. As a result, we have now seen an Executive Order on Improving the Nation’s Cybersecurity from the US government as well as the Cyber Resilience Act from the EU.
Supply chain security is a top priority for DevOps teams. Katie Norton, Senior Research Analyst at IDC, recently highlighted that “organizations are less confident in the security of their software supply chain and must grow their adoption of practices and tools that secure it.” In her view, “every organization building software is now a target for a software supply chain attack, and they need to be diligent to avoid being the next victim of a high-profile security breach.”*
Software supply chain security provides transparent, verifiable proof that software and how it was produced are free from compromise and tampering, enabling software consumers to independently verify whether the software meets their criteria for usage.
In this article, we will discuss supply chains and attacks, best practices to secure the software supply chain, how these concepts apply to mainframe applications, and even a new technology that will help secure the mainframe software supply chain!
What is a supply chain attack?
Those of a certain age may remember there was a time when you could open a bottle and it didn’t have anything between the lid and the container; otherwise known as a safety seal. The history behind this is both tragic and interesting. In 1982, someone intentionally laced Tylenol with poison resulting in the death of 6 people. In addition to the human cost, you can imagine the economic ones as panic ensued and recalls became necessary. Essentially, the supply chain between the producer and consumer had become compromised and the public lost confidence in the product. Johnson & Johnson needed to act fast to restore that confidence or people would quickly begin to avoid using Tylenol.
The solution was to add the safety seal so that consumers would know if something was being tampered with. The bottle itself detailed the ingredients and the safety seal was added to ensure they would know if something that was not in the intended ingredients list was added. What does this have to do with software? Believe it or not, there are direct correlations.
Software also has a Supply Chain
Just like Tylenol, the applications we rely on every day, directly and indirectly through the organizations we purchase goods and services from, have a supply chain. Whether it’s third-party open source or proprietary libraries or the critical compilers and tools we use to build and deploy them, all software is created from a list of key ingredients, just like real-world products.
Very often these ingredients are not obvious to the consumer and so they have no way of knowing if any of them have been compromised. Bad actors know this and so a recent trend has been to move the attacks away from the applications themselves towards the processes, tools, and ingredients that create the applications. You’ve likely heard about several high-profile examples of this. Log4j, SolarWinds, and the Kesaya ransomware attack are all examples of supply chain compromises. In each case, some part of the supply chain for those products was compromised allowing the perpetrators to amplify the amount of damage being done. The problem is so widespread, governments around the world are getting involved and recommending new standards be adopted and even mandating they be supported by their vendors. So what is it they are recommending? It should be no surprise that just like real-world supply chains, the solution lies in being transparent with the list of ingredients that make up the application and ensuring that it is evident when that list has been compromised.
Taking Inspiration from Real World Supply Chains
When faced with a supply chain crisis, Johnson and Johnson moved to create tamper-evident packaging. Likewise, in the world of software, two key concepts have emerged. The first is the Software Bill of Materials or SBOM we previously mentioned. The second is a cryptographic wrapper called an Attestation that indicates if the event or artifact it wraps can be trusted and indicates if it has been tampered with.
An SBOM is a list of key ingredients that make up an application and the various attributes describing them. It can account for open source or proprietary software libraries that are in use, along with their producers and version, or things like compilers used to build the software or tooling that was used in its development. It will also contain the supplier’s name.
By using this information along with the fact that the SBOM is digitally signed, a customer can verify that the author of the software is legitimate and trusted. There are a few formats in use such as CycloneDX or SPDX but the concept is the same regardless. Just like an ingredients list on consumer packaging, its purpose is to transparently let people know what is in the product they are consuming so they can make decisions accordingly. Allergic to a certain ingredient? You can avoid the product. No tolerance for known vulnerabilities? You can decide not to install or work with the vendor to ensure the vulnerability is addressed before rolling out updates. Similar to the seal of consumer packaging, attestations are formats that apply a signed wrapper that can be checked to ensure the data in the SBOM or other artifact or event being wrapped has not been modified.
How Does This Apply to Mainframe?
Mainframe applications are not immune from supply chain attacks and neither is software produced for only internal consumption. Applying the same standards and practices that are being adopted in response to supply chain attacks in the world of cloud, mobile, and distributed development makes sense. By using the same processes regardless of platform, organizations can ensure that software being deployed or used is held to a single standard across the enterprise, without exception. Likewise, the software you produce for your own customers should be considered just as vulnerable as commercial-off-the-shelf products built for wide-scale distribution. You may think, “we don’t use open source or third-party libraries for software we build for internal use so this doesn’t apply” but think about it — aren’t the tools, compilers, and runtime libraries for subsystems like CICS or IMS third-party ingredients to your applications? Wouldn’t you want to know if a z/OS application component has a known vulnerability before you rely on it to provide critical business functionality?
Endevor Adds SBOM Support to Secure the Software Supply Chain for Mainframe Apps
Broadcom’s leading SCM for mainframe native pipelines, Endevor, has made keeping up with the ever-evolving world of software security that much easier by adding the ability to create SBOMs as part of your Endevor automation. We’ve done this by introducing sbomz, an SBOM generation tool for z/OS. It is a command line tool that runs on z/OS UNIX and is capable of generating SBOMs from Endevor inventory locations, Endevor packages, z/OS files and data sets, and Endevor Team Build projects for pipelines built around Git.
Sbomz supports digitally signing SBOMs so that consumers can independently validate the signature to confirm that the SBOM can be trusted. It supports industry-standard SPDX and CycloneDX SBOM formats.
In the following video, we demonstrate how Endevor’s sbomz utility can be used to generate signed SBOMs from an Endevor processor. Subsequently, these SBOMs can be checked by tools like Open Policy Agent to ensure the software about to be delivered has not been compromised, either maliciously or by accident.
Industry Standards and Formats Matter
Another consideration for adopting these evolving standards beyond proving compliance to a single set of organizational policies is the world of tooling that it opens up. Ensuring mainframe apps are built with the same formats and standards for SBOMs and attestations means you can use the same policy governance tools for your mainframe pipelines. This aligns with Broadcom Mainframe’s Open-First approach and how it enables various Enterprise tools and processes to integrate with the mainframe.
By integrating with the same policy governance tools, this data becomes readily consumable to additional stakeholders in your organization such as AppDev Owners, CISOs, etc. For example, you can load your SBOMs in OWASP’s Dependency Track to easily see your exposures to vulnerabilities when they are reported. Or you can use tools like Grype to check for new vulnerabilities before deploying. In fact, that last one leads to another key benefit of adopting supply chain security practices.
Supply Chain Security + DevOps = Policy Automation
Software security is nothing new — though it is continually evolving. For instance, it’s common practice to use static application security testing tools to identify potential security problems in code before it’s even built. Organizations might have policies in place to dictate that these scans must be run and perhaps it’s even built into the pipelines. Other organizations might have pre-deployment meetings or checklists to ensure these steps don’t get missed. To make things bulletproof though — these policies should be automatically checked continuously as the software gets built and ultimately deployed. This practice is known as policy automation.
Tools like Open Policy Agent let you encode your policy as snippets of code that you then store in your version control system shown as the “Policy Rules” in the diagram above. You can lock that repository down and apply the same standards of control to ensure they are only updated by authorized people for known reasons, with versioning and audit history stored along the way. By using the policy agent to check the tamper-proof evidence you are producing as part of your pipeline, you are continually guarding against compromise. That evidence can be stored in an Enterprise “Evidence Store” that can also give your auditors a one-stop shop for ensuring that policy compliance is being adhered to.
It is also important to note that the SBOM moves with the software through the supply chain. Once all organizations are complying with the guidelines from the US and EU governments, you should assume/expect/require an SBOM for any software you consume and provide an SBOM for any software you create without exception.
Next Steps
I would recommend familiarizing yourself with the sbomz utility by reviewing our documentation. It has tutorials, recommendations, and samples of how to use sbomz in processors, for packages, for data sets, and more. Once you begin creating SBOMs for your mainframe applications, identify integration opportunities within your organization — perhaps you are leveraging tools like Open Policy Agent or OWASP’s Dependency Track for other applications. These same tools can be leveraged for mainframe applications. If you are interested in seeing sbomz in action with one of these sample tools, comment below — the next blog is right around the corner! And as always, we’re happy to consult with you to see how this can apply to your Endevor configuration.
If you’ve found this blog useful and would like to learn more about modernizing your mainframe development practices, you can do so at our Modern Mainframe blog site where we talk about integrating mainframe into Enterprise toolchains with practical examples and use cases.
*Source: IDC, How Confident Are Organizations in the Security of Their Software Supply Chain, and What Practices Have They Implemented to Secure It?, Doc #US49381823, Jan 2023
