CA SYSVIEW Performance Management: Crossing the Threshold to Modern DevOps Processes
A prior tech blog outlined how the CA SYSVIEW development team implemented more robust DevOps tooling using CA Endevor and a fully automated batch regression testing infrastructure. Seeing the benefits reaped from that effort, the team drove forward to augment their DevOps processes by implementing a number of industry-standard tools that increased testing and yielded other benefits while saving human effort.
Testing enhanced using Python, GitHub, Jenkins and Zowe
One of the key enhancements delivered by the SYSVIEW team in Version 16.0 is the CA SYSVIEW REST API, a facility providing endpoints that access data and actions via SYSVIEW’s Command Line Interface (CLI). While developing the test plan for this exciting new facility, the team determined that some tools not typically associated with mainframe R&D could be leveraged: Python and Jenkins. They set about authoring Python scripts managed in GitHub Enterprise to exercise the REST API. Over time, scripts were written to test every SYSVIEW primary command, with automated test execution driven by Jenkins three times a week. Executing the larger, more complex sets of commands via Python scripts exposed some memory and threading issues when multiple tests were run in parallel. Python-driven testing also found a problem in a 3rd-party vendor’s code, encountered only when the Python script and SYSVIEW’s own scheduling facility issued several thousand JVM commands against a single JVM without restarting it.
Other Version 16.0 deliverables are SYSVIEW support for the Zowe API Mediation Layer and the CA SYSVIEW Performance Management Plug-in for Zowe , which lets application developers, DevOps and CI/CD administrators, and other IT roles issue SYSVIEW commands. As with the Python scripts, the team developed tests of all SYSVIEW commands using the plug-in for Zowe, expanding the overall test suite 30% and fully exercising SYSVIEW code with automated test runs twice weekly.
The Python scripts and SYSVIEW plug-in for Zowe provide critical tools for automating tests, but these methods require human intervention either to manually initiate execution of the test suite or to schedule a test run.
An automated CI/CD pipeline for mainframe is developed
The team crossed the threshold to modern DevOps processes by developing a CI/CD pipeline that automatically detects new code ready for testing, deploys to a test system, executes automated testing, and reports test results to the team, all with no human involvement. To do so, the team leveraged CA Endevor Bridge for Git and Zowe, both of which are solutions used by Broadcom customers, along with other modern DevOps tools.
It all starts when a developer promotes a code change to “ready for QA” status in Endevor, initiating pipeline execution as follows (follow along in the diagram below):
· CA Mainframe Webhook Server detects the change and pushes a change event notification to CA Endevor Bridge for Git.
· Endevor Bridge for Git relays the changes to GitHub.
· A web hook deployed in GitHub triggers a Jenkins pipeline.
· Jenkins invokes a Python script to execute native Zowe commands to copy updated code elements to a test system, and cycles SYSVIEW if necessary to activate new code.
· Leveraging native Zowe commands and the CA SYSVIEW Performance Management Plug-in for Zowe, Jenkins executes a series of test suites and scans, including a security vulnerability scan.
· Jenkins emails a full report of test results to the team for analysis.
Again, no manual steps are required from the time the developer promotes the new code to QA ready status to the time the test results show up in the developer’s inbox.
Of course, where it may not be desirable or feasible to execute the pipeline on every event, it is possible to implement delays, or even schedule periodic pipeline execution. In fact, in addition to utilizing automatic execution, the SYSVIEW team has scheduled pipeline runs to occur on a nightly basis. The nightly test runs are yielding value. One developer noted, “I updated the MQPAGE command and field table and it was nice that the nightly scan caught that two field descriptions were missing. So good catch!”
The pipeline was designed to be portable for use with any code base. Execution parameters can be modified to specify which libraries are copied, which source and destination systems are used, and so on. Additionally, any Endevor event type can be set to trigger execution of the pipeline.
Implementation cost
Even while learning new languages and tools “on the job”, the team was able to complete the automated Python script-based testing process and test cases in parallel with development of the SYSVIEW REST API functionality. Developing automated testing with the SYSVIEW Performance Management Plug-in for Zowe was a straightforward effort. In total, both methods yielded over 2,000 additional automated test cases with only a few months effort. A 26-year z/OS professional noted, “The Zowe plug-in makes it easy for even a beginner to learn and create scripts and tests. I am still learning Python, but while creating and testing scripts the messages are clear about problems and how to fix them.”
Development of the automated CI/CD pipeline was completed as a process improvement initiative by two members of the SYSVIEW development team. The pipeline prototype, including researching tools and collaborating on enhancements to Endevor Bridge for Git with the development team, was completed in about a month. It took just another few months of part-time effort to complete the pipeline used by the team today.
Benefits realized
As well as being versatile additions to the QA tool chest, the Python script and Zowe plug-in based testing methods save manual testing time and directly improve SYSVIEW quality by exercising code using multiple interfaces. The ability to execute complex scripts allows for testing of previously impossible scenarios, resulting in exposure of problems triggered only on certain commands or displays. Some of the problems found were in long-existing commands, and were not evident when comparable tests were executed in batch mode or using the online interface.
The CI/CD pipeline provides a fast, effortless means to test new code using modern DevOps tooling. Automatic kickoff of SYSVIEW testing based on Endevor events was previously not possible. Zowe provided the necessary technology to automate mainframe activities in the pipeline as directed by Jenkins. Both automatic and scheduled daily pipeline execution drive a “fail fast” process, resulting in problems being fixed early, before additional code complicates the remedy.
A key benefit of leveraging Endevor Bridge for Git is that source code in GitHub is maintained in 100% synchronization with the source in Endevor. In addition, GitHub’s code merge capability enables parallel development of a given code element by multiple developers.
Key insights
The drive to engineering excellence is a journey, and mainframe development teams can go a long way toward implementing innovative DevOps practices using readily available tools. The modern DevOps tools leveraged by the SYSVIEW team have enabled far more sophisticated test automation requiring far less human involvement. The benefits include higher quality deliverables, faster time to market, more team capacity for innovation, and greater job satisfaction for team members.
