Adding SBOM Generation and Hash Validation to Endevor Package Ship Facility
Using Endevor sbomz utility to generate/sign SBOMs for Endevor Package and z/OS Cryptography Services for validating Hash246 values.
OOn May 12, 2021 The United States of America issued The President’s Executive Order (EO) 14028 on Improving the Nation’s Cybersecurity.
Section 4 . Enhancing Software Supply Chain Security establishes the issue of standards, procedures, or criteria that enhance the security of the software supply chain in order to secure software development environments. The action regarding the monitoring operations and alerts and responding to attempted and actual cyber incidents, states that a Software Bill of Materials (SBOM) for each product must be provided to the purchaser directly or published on a public website.
TThe EU Cyber Resilience Act (CRA), proposed on September 15, 2022 by the European Commission, is the first EU-wide legislation addressing cybersecurity requirements for software and hardware manufacturers and developers with digital elements connecting to the internet. In contrast to the U.S. Executive Order, the CRA extends to all vendors who place their products on the EU market.
SBOMs are a requirement of the proposed European Union (EU) Cyber Resilience Act. This legislation will require all software vendors to document materials used in software development with an SBOM to promote software supply chain security (SSCS), software vulnerability remediation, and license compliance.
The European Union Agency for Cybersecurity (ENISA) has published CRA Requirements Standards Mapping. In the Vulnerability handling requirements, the mapping establishes:
- All libraries and external components used in the software part of a product, including their version number should be listed in a SBOM available to the user
- Such SBOM should be compliant to the relevant standards (e.g., ISO/IEC 5921:2021 also known as SPDX [2] or CycloneDX [4] standard)
Both Executive Order 14028 and the EU Cyber Resilience Act establish the mandatory use of SBOM in the software supply chain. In the Medium article embedded below, which was written by my teammates Michael Bauer and Vaughn Marshall, you will find practical and clear information about the Software Supply Chain and SBOM concepts.
SBOMs implementation helps organizations and developers to have constant visibility into the SDLC. Although open source software is the genesis of modern software development, it can be a major contributing factor in software vulnerabilities.
SBOMs are particularly important for projects using open-source components, which, according to research by Gitnux is more than 96% of applications!
SBOMs play a pivotal role in mitigating supply chain risks by giving developers and stakeholders greater visibility into the software components they rely on. By analyzing the SBOM, organizations can identify known vulnerabilities, track and manage their software usage, and ensure compliance with licensing agreements by providing a comprehensive list of all components and licenses.
SBOM creation and validation process can be tie directly into the continuous integration and continuous deployment (CI/CD) pipeline at the deploy stage. If the SBOM signature cannot be validated and if the hash values of the deployed components do not match, deployments should fail.
TThe type of SBOM generated could be categorized as a Build SBOM according to CISA (Cybersecurity and Infrastructure Security Agency) which describes a Build SBOM as an “SBOM generated through the analysis of artifacts (e.g., executables, packages, containers, and virtual machine images) after its Build, and such analysis generally requires a variety of heuristics. In some contexts, this may also be referred to as a “3rd party” SBOM.”
Build SBOM provides greater confidence that the SBOM representation of the product artifact is accurate due to information available during Continuous Integration/Continuous Deployment (CI/CD) processes. A Build SBOM provides visibility into more components than just source code. Another advantage to automating Build SBOM creation as part of software delivery workflows is the ability to digitally sign your SBOM to establish trust with your clients by confidently attesting to what’s in your software.
Based on the information described above, a few questions arise regarding software delivery processes in the Mainframe world and compliance with Executive Order 14028 / EU CRA:
- Could we generate SBOMs during the execution of deployment and delivery tasks on the Mainframe?
- How could these SBOMs be generated?
- Could SBOMs be signed and their authenticity subsequently validated?
- Could the Hashes of the deployed components be validated against the information included in the SBOM by the generation process?
The answer to the three first questions is Broadcom’s sbomz utility.
sbomz is a z/OS UNIX-based software bill of materials generation utility that enables SBOMs generation for Endevor and Endevor Team Build applications, providing a z/OS UNIX CLI-based tool for z/OS generation of SBOMs from the following sources:
- Endevor inventory location
- Endevor package
- Team Build project
- z/OS PDS(e), PS, or z/OS UNIX files
The answer to the fourth question is ‘Yes’ and can be achieved using z/OS shell command sha256. This shell command calculates and checks SHA-256 cryptographic hashes and requires ICSF installed and running because it uses the ICSF One-Way Hash Generate callable service.
This article describes a use case that shows how to add SBOM generation and Hash validation to Endevor Package Ship Facility, with the objective of satisfying the requirements established by the Executive Order 14028 and the EU Cyber Resilience Act; as well as, to guarantee the validity of the SBOM and the hashes of the elements delivered during the CD phase in the DevOps life cycle.
No modifications are needed to the Endevor processors or User Exits, it would only be necessary to adapt the corresponding skeletons, if the shipping option is executed under ISPF, and adjust the modeling members accordingly to the transmission method used.
FFor more information about Endevor Package Ship Facility, take a look to this blog Package Shipping and Post-Ship Scripts De-mystified written by my colleague Joseph Walther in the Endevor Community. He has also been involved in the improvement of the approach described in this article.
The image below shows “the big picture” of the approach implemented:
The endevor shipping process involves a Host Site(where the running Endevor instance is located) and one or more Remote Sites. This article describes the process between a Host Site and a single Remote Site.
In the northbound or upstream edge (Host Site), the main phases executed are: Build Shipment Request and Host Package Shipment job execution. Once the Host Package Shipment Job is executed, the Remote Package Shipment JCL is submitted to the southbound or downstream edge (Remote Site) and a transmission confirmation condition is triggered to Endevor Package component. The Remote Package Shipment Job will confirm the result of the Remote JOB execution, sending back a confirmation JCL to the the Endevor instance at the Host Site.
When the confirmation Job is run at the Host Site, the final status of the package shipping execution on the Remote Site is updated in the Endevor package file.
The components needed on each of the edges included in the image included above as well as each of the three main phases with their respective steps are described in the next sections.
✔️Required Software & Configuration on each Site.
💠Host Site
💡 Endevor® Version 19.0 up-to-date.
💡 Endevor® Web Services (optional)
EEndevor Web Services component is required if you want to execute the shipping process using the Endevor Plug-in for Zowe CLI, a REST API call using a client application or Endevor Plug-in for Eclipse.
The Endevor ISPF application is the user interface selected to show the implementation of the approach described in this article.
💡 sbomz utility.
SThe Sbomz utility on z/OS is installed by applying Endevor® 19.0 PTF LU08967 and all subsequent enhancements provided on Broadcom Support Portal. Then, sbomz have to be deployed from the SMP/E installation libraries by using the sample jcl ENSBOMZI in the CSIQJCL data set:
<smpehlq>.CSIQJCL(ENSBOMZI)Follow the instructions within the sample jcl to complete the procedure.
💡 Public and Private key pair generation to digitally sign SBOMs.
SIt is necessary to create a Public/Private key pair, in order to digitally sign the SBOMs at the Host Site using the private key to later validate their authenticity in the Remote Site using the public key.
Sbomz provides a command which generates a public and private key pair using the ED25519 algorithm:
sbomz sign key-gen💠Remote Site
💡 sbomz utility.
RRemote Site does not require an Endevor Instance installed. Therefore, the installation of sbomz utility at the Remote Site must be done using the binary file ENSBOMZ included in the Endevor Target Library <hlq>.CSIQHFS at the Host Site. Follow the instructions in the sbomz installation manual with the aim of configuring the USS environment for sbomz execution.
💡 IBM Integrated Cryptographic Services Facility (ICSF)
IIn order to validate the cryptographic hashes of the files received at the Remote Site, the USS shell command sha256 is used.
Additional information about sha256 shell command can be found in the z/OS Unix System Services Command Reference guide.
💡 Public key to verify SBOMs signature.
The public key generated using sbomz utility at the Host Site must be copied to the Remote site for SBOM authenticity validation.
✔️SBOM generation and Hash validation Use Case — Main Phases.
💠Host Site
🕐 Build Shipment Request Phase
1️⃣ In this step, the package ship request is built using the Endevor IDE available on the z/OS image or a batch JOB: ISPF Package Foreground Options Panel, Batch Package Facility, or through any Client Application like endevor plug-in for zowe CLI , Endevor Eclipse-Based UI, REST API call from a REST Client Application.
The interface selected to show this use case is the ISPF Package Foreground Options which can be executed from Endevor’s native ISPF interface or from QuickEdit.
When the shipment queue is submitted, Endevor builds and submits a JCL stream to ship the packages. The Endevor ISPF application uses the SHIP Skeletons included in the CSIQSENU library to construct the Host Package Shipment JCL that will trigger the shipping process. The skeletons mainly used are:
Suffix xxx in C1BMXxxx, represents the selected transmission method: COM (XCOM), FTP (IBM FTP), BDT/BD1 (Bulk Data Transfer Jes & version 2), NDM (Connect:Direct) and LOC (Local Destination using IEBCOPY).
The skeleton that contains all the JOB steps for the generation and signing of the package SBOM is embedded in the skeleton corresponding to the selected transmission method.
To ensure security and traceability, the package SBOM must be created under the Endevor Alternate ID. By enabling the PACKAGE_SHIP_WITH_ALTID = ON option in the ENCOPTBL table, Endevor will use the alternate ID, instead of the user ID, when submitting a package ship request.
🕑 Host Package Shipment Job execution phase
This phase executes the JCL built during Build Shipment Request Phase which includes the additional steps for the SBOM management. The JOB runs under Endevor Alternate ID:
2️⃣ The Ship Utility C1BMX000 builds a shipment consisting of the following:
🛳️ A staging data set for each library involved in the package execution.
🛳️ JCL for execution at the remote site (AHJOB), consisting of a SBOM Sign Validation step, a SBOM Hash extract step, a Hash validation step, an IDCAMS (delete) step, and confirmation steps.
🛳️ Data transmission utility commands and/or JCL to transmit the data sets and execute the remote copy/delete job (Not applicable for local transmissions). The utility uses the primary model members #PSNxxxs located in the CSIQOPTN library, where ‘xxx’ represents the selected transmission method ‘COM (XCOM), FTP (IBM FTP), BDT/BD1 (Bulk Data Transfer Jes & version 2), NDM (Connect:Direct), LOC (Local Destination using IEBCOPY)’ and ‘s’ is a one character suffix (alphanumeric or national character).
🛳️ A file of correspondences between host and remote production and staging data sets.
🛳️ Complementary files and JCL, if necessary.
The ship utility then populates the staging data sets, records the shipment, and passes the entire shipment to the next step.
C1BMX000 utility uses the primary model members located in the CSIQOPTN library
3️⃣ By default, ‘sbomz endevor sbom’ command runs the Endevor batch SCL processor in the foreground to extract information from an Endevor Package, using any of these request: list package id and list package action.
If your Endevor configuration prevents sbomz from automatically extracting the CSVs and reports, ‘sbomz endevor sbom’ command provides options to specify previously generated CSVs and report files.
In this approach, the CSVs files are generated in advance using CSV utility (BC1PCSV0) before the Package SBOM generation Step:
if sbomz cannot derive output artifacts from Endevor components and Endevor MONITOR COMPONENTS capabilities is not used, the generic data set and file options ‘— generic-file-source’ could be used to include specific components files in your SBOM.
A REXX code is used to get all the components from the Package Backout entries:
4️⃣ The Package SBOM is built from the CSVs files generated in the LSTPKGID Job step and the — generic-file-source commands produced by the execution of the CMPNTGEN Job Step.
Sbomz is compiled using the IBM Enhanced ASCII option, which enables sbomz to produce ASCII files and output. Therefore, the SBOM constructed in the Standard Output File pointed by STDOUT ddname will be in ASCII format. Also, CSVs and text files used by the sbomz utility must be converted from EBCDIC to ASCII format before executing sbomz.
Additional information about the ‘sbomz endevor sbom’ command can be found in the Endevor Reference Guide — sbomz CLI reference section.
5️⃣ Once the SBOM is built, it’s time to sign it using the private key generated during the configuration task ‘✔️Required Software & Configuration on each Site’
6️⃣ The signed SBOM is added as type SBOM to an inventory location created in one of the environment available for Endevor artifacts (ADM Environment in this use case). Before adding the SBOM to the Endevor location, it must be converted from ASCII to EBCDIC format.
The package SBOM is now ready to be transferred to the Remote Site using the Transmission Method selected during the Build Shipment Phase. In this use case, FTP was the chosen method :
If the transmission of the SBOM to the REMOTE SITE is successful, the Job Step (or secondary job depending on the transmission method) that transmits all the necessary files to the remote site is run.
The last action executed during the Host Package Shipment JOB (or secondary JOB if there is one) is the SUBMIT to the Remote Site of the jcl contained in the AHJOB data set. This trigger the execution of the Remote Package Shipment Job at the Remote Site.
💠Remote Site
🕒 Remote Package Shipment Job Execution Phase
7️⃣ The first job step that is executed in the Remote Package Shipment job verifies that the SBOM signature is correct. To do this, the public key that was copied during the configuration of the SBOMZ utility on the Remote Site is used. If any inconsistency is detected during the signature verification, the Remote JOB will notify the Endevor instance on the Host Site about the issue by sending the NOTOK confirmation JCL (embedded in the JCL of the Remote Job).
8️⃣ This job step extracts from the SBOM the hashes of all the Host Staging data sets that were copied to the corresponding Remote Staging data sets during the previous phase. The JOB step executes a REXX which processes the SBOM (Json File) using the z/OS JSON parser.
The hash.out file contains all the Remote Staging data sets elements and their expected hash values got from the Host Staging data sets entries in the SBOM package:
9️⃣ The HASH256 Step computes the hashes of all the artifacts in the Remote Staging data sets and compares them against the Host Staging data sets hashes gotten from the SBOM during the execution of the previous step. If any mismatch is detected, the Remote JOB will notify the Endevor instance on the Host Site about the issue by sending the NOTOK confirmation JCL (embedded in the JCL of the Remote Job).
CHKHASH Job step computes Remote Staging data sets hashes and validates them against their corresponding Host Staging data sets hashes in the hash.out file.
🔟 If no issues were detected in both the SBOM signature verification step and the remote storage dataset hash validation step, the elements are copied from the Remote Staging data sets to the Production Libraries.
If any issue is detected during the copy step, the Remote JOB notifies the Endevor instance on the Host Site about the issue by sending the NOTOK confirmation JCL.
In the event that no problem has been detected during the copy step, the Remote JOB notifies the result to the Endevor instance on the Host Site by sending the OK confirmation JCL. In this case, the confirmation JCL also carries a JOB step that is responsible for moving the SBOM from Stage 1 to Stage 2 in the ADM Environment.
✔️Additional Resources.
🔮 Skeletons, Source Code and Primary Models Members
ISPF skeletons , the primary model members and the source code of the developed programs are currently in the debugging/optimization phase.
As soon as they are finished, they will be made publicly available in the Endevor repository on github and the respective url will be added to this article. So, stay tuned!
