How mainframe engineers can leverage CA SYSVIEW Zowe CLI Plugin in their CI/CD pipelines
{Ecosystem} In this blog, we will take a deeper dive into how mainframe developers and systems programmers can leverage the CA SYSVIEW CLI plug-in for The Open Mainframe Project’s Zowe Command Line Interface(CLI) to develop CI/CD test suites for validating their products and/or systems.
We are building upon a prior blog, where we detailed how users can access z/OS performance data and resources (data sets, subsystems, regions, etc.) using the CA SYSVIEW Performance Management Zowe CLI plug-in. As well as a subsequent blog, where we detailed the benefits realized in implementing a CI/CD pipeline in our CA SYSVIEW development environment.
The sample test program I’m detailing can be used by developers and system administrators. This program can be used to ensure all data sets created under a SMS Storage Group are encrypted. This Storage Group might be assigned to a product team generating sensitive data that must be protected.
Challenges
One of the primary challenges of developing test suites in a mainframe CI/CD pipeline is the fact that modern DevOps tools do not integrate well with the z/OS ecosystem. Creating modern test suites is not an easy feat. Thanks to Zowe and the Open Mainframe Project, z/OS accessibility pains are a thing of the past.
The next set of challenges arise when deciding how to develop programs to access and test the various components of z/OS, applying modern techniques. CA SYSVIEW can help with that!
Due to CA SYSVIEW’s versatility, modern test suites can be developed using the CA SYSVIEW Zowe CLI plug-in to test the various z/OS components and its subsystems: CICS, IMS, MQ, and DB2, as well as TCPIP, JVM, USS, DATACOM, and IDMS.
Configure Jenkins
Before executing the CA SYSVIEW Zowe CLI Python test program, check with your Jenkins admin to ensure the latest version of Zowe and Python3 are installed on your Jenkins agent, where the pipeline job will run. In addition, perform the following tasks:
Required: Install the latest version of the CA SYSVIEW Zowe CLI plug-in. From your Groovy script, issue:
sh "zowe plugins install @broadcom/sysview-for-zowe-cli@zowe-v1-lts"
Recommended: Create your CA SYSVIEW profile to use throughout the script when invoking CA SYSVIEW Zowe CLI. Otherwise, you will have to specify the parameters on every invocation.
Invoke the Test Scripts
The test script is written in an Object Oriented fashion in Python. Starting the tests from Jenkins is as simple as issuing the following command:
sh "python3 -m main storageGroup"
Where main is the Python test script and storageGroup is the name of a SMS Storage Group. The Storage Group will be interrogated to detect whether any unencrypted data sets exist on the volumes within the group.
Driver program, main.py
main.py is that sample driver program responsible for invoking the various test scripts created by the team. In this sample, only one test is created and invoked.
The sample driver program is very simple. It creates a Detect_Unencrypted() test object passing the Storage Group requested by the invoking user, then executes the test.
Note, there’s no input validation or thorough error handling. This is intentional to keep the sample short and concise. In practice, I encourage you to add input parameter validation and error handling.
Before diving into the Detect_Unencrypted() class, we need to talk about the Test() base class. Detect_Unencrypted() and all future tests should inherit from the Test() class.
The Test Design
I encourage you to leverage Python’s OOP capability. In this sample, defining base and children classes is excessive, but in practice, it is extremely beneficial in deploying new tests quickly and uniformly.
The Test() Parent Class
This is the parent class that all tests inherit from. Although it is not required, it greatly reduces coding and coding errors between tests by eliminating code duplication.
The Detect_Unencrypted() Child Class
__init__()
The first part of this class is defining the class constructor. It initializes the class variables and invokes the base class constructor.
Method run()
run() is a public method that orchestrates the execution of the test. It invokes private method getStorageGroupInfo() to obtain information about the SMS Storage Group requested by the user.
Upon successful retrieval of Storage Group data, private method findUnencryptedDatasets() is invoked to find all unencrypted data sets, if any, allocated on all volumes within the user’s provided Storage Group.
Method getStorageGroupInfo()
getStorageGroupInfo() is a private method that retrieves information about the SMS Storage Group provided by the user. It does so by leveraging the CA SYSVIEW Zowe CLI plug-in to gather the Storage Group information on z/OS and returns it to the program in JSON format.
Variable sysview_cmd contains the CA SYSVIEW Zowe command string. It consists of:
- The CA SYSVIEW Zowe CLI request
zowe sysview display
- The CA SYSVIEW command string
smsvols;select group eq self.storageGroup and mvs-stat eq online
This is the equivalent of issuing the following commands on the CA SYSVIEW z/OS 3270 Interface:
- SMSVOLS
To display all SMS volumes defined to the active SMS configuration. - SELECT GROUP EQ storageGroup AND MVS-STAT EQ ONLINE
To select/filter the Storage Group requested and return ONLINE volumes only.
Pro Tip: Use SET SELECTNEXT when possible, as shown below, to pre-define the selection criteria before executing the CA SYSVIEW primary command. This method is almost always more efficient than issuing the primary command then specifying the selection criteria.
set selectnext 'group eq self.storageGroup and mvs-stat eq online'; smsvols
- The CA SYSVIEW Zowe CLI options
--all-rows --rfj --fields Volser --sysview-profile sysv_prof
Those options instruct CA SYSVIEW to return all volumes VOLSERs that pass the selection criteria in JSON format, using profile sysv_prof parameters that we defined earlier in the document.
The command response is returned as a JSON object. We check if any rows, which represent volumes, are returned by the command. If none, that’s an indication that there are no volumes define to the Storage Class.
Method findUnencryptedDatasets()
This private method takes in the list of volumes in JSON format, iterating through the list and examining the Volume Table of Content (VTOC) of each volume in the list and identifying datasets that do not have an encryption label.
Variable sysview_cmd contains the CA SYSVIEW Zowe command string. It consists of:
- The CA SYSVIEW Zowe CLI request
zowe sysview display
- The CA SYSVIEW command string
vtoc volume;select encrypted b
This is the equivalent of issuing the following commands on the CA SYSVIEW z/OS 3270 Interface:
- VTOC volume
To display the VTOC content of the specified volume. - SELECT Encrypted B
To select/filter all data sets that have the encrypted field set to blank
Using the Pro Tip mentioned prior, the command string would be:
set selectnext 'encrypted b'; vtoc volume
- The CA SYSVIEW Zowe CLI options
--all-rows --rfj --fields DatasetName --sysview-profile sysv_prof
Those options instruct CA SYSVIEW to return all data set names that pass the selection criteria in JSON format using profile sysv_prof parameters that we defined earlier in the document.
Results
If unencrypted data sets were detected within the Storage Group
2021-04-23 14:49--> Encryption test is in progress..
USER1.DATASET1
USER2.DATASET6
Uh oh! Unencrypted data sets detected within SGGROUP1If unencrypted data sets were not detected within the Storage Group
2021-04-23 17:22--> Encryption test is in progress..
Great! No unencrypted data sets exist within SGGROUP2The results can be written to a file that is attached and emailed via Jenkins instead of printed to standard output.
Next Steps
We value user feedback and are always looking to improve our software. Try out the new CA SYSVIEW for Zowe CLI plug-in today and let us know what you think! Please send any questions or comments related to CA SYSVIEW to support@broadcom.com and also visit this Broadcom site;
Finding out more
If you enjoyed this blog checkout more Zowe blogs here. Or, ask a question and join the conversation on the Open Mainframe Project Slack Channel #Zowe-dev, #Zowe-user or #Zowe-onboarding. If this is your first time using the OMP slack channel register here.
Zowe is owned and managed by the Open Mainframe Project which is a Linux Foundation project.
