Batch Performance Test on Commit with CA MAT — 3 steps
Wouldn’t you prefer to understand your mainframe batch application performance with every code change? Is performance analysis part of your mainframe batch application testing today? Weekly, monthly, maybe only quarterly? And how do you handle the results? Do the people that check performance actually know the application code? Or do they manage the whole system, spending hours to isolate a problem and later more hours trying to guide developers, who can actually fix the application?
I am excited to announce a new Performance on Commit capability of CA Mainframe Application Tuner (MAT) that is available with the new release of CA MAT version 12. It can help you save a tremendous amount of critical application developer and system programmer time by executing performance tests on every commit, meaning on every application change.
This performance analysis with every commit, incorporated with DevOps best practices of making small changes brings performance analysis up front in the change process solving problems early, long before production. For example, the study below shows that defects found early at the functional testing stage can save up to 64 times the cost of defect resolution as compared to finding them later in the application life cycle.
Industry experts suggest you test your performance early and often. This sounds great, but how to actually go forward with adding performance testing into your regular development process for a mainframe team? Performance should be part of the very mindset of the project and the team!
Diego Lo Giudice, Forrester VP: “Make developers stewards of the code they write, and regularly test performance on smaller pieces of code to ensure compiled code performance.”
New “CA MAT Detect” and “CA MAT Analyze” plugins for Zowe CLI provide open interfaces for modern CI/CD orchestration tools. This enables testing suites to integrate performance checks into automated pipelines and your quality certification procedures without manual development team effort. Modern frameworks for Mainframe, such as ZOWE, make this integration easy and allow mainframe automation to behave the same as working with distributed systems. Let’s take a closer look…
1. Performance on Commit: FIRST STEP — AWARENESS
The first step is to ensure your development team understands the application performance before any changes are made — that is to collect a baseline of performance metrics such as CPU consumption, Elapsed Time, EXCP count for System IO, and SRVU counts for a specific job. A simple document or email with these metrics is enough. This would clearly outline for each member of the team that their code affects not only functionality, but performance and allow them to understand the effects particular changes might have on the way an application performs.
Gathering this baseline is easy with the new Zowe “CA MAT Detect” plugin. You can easily retrieve the base metrics from the testing system via zowe pma get perf command and automatically send them in an email after each commit.
In the example below an email is sent to the development team, showing base performance metrics for the batch job DEMOCICD, which runs the COBOL application DEMOCOB1. The deviations of the metrics from the average values after this latest test run are highlighted in the boxes upfront.
2. Performance on Commit: SECOND STEP — DETECT and ALERT
The next step is to define thresholds and automatic checks for deviations to be used as part of your automation pipeline. The pipeline raises an alert when the application exceeds any of the defined thresholds, thus prompting the developers to immediately address the detected issue.
However, how do you determine a threshold and how do you set up automatic alerting that is clever enough to notify only when performance deviation is not a fluctuation?
With “CA MAT Detect” you don’t even need to set up any thresholds. The silent backend of “CA MAT Detect” called Performance Management Assistant is able to learn your application and with built-in intelligence to only notify users when the performance deviation really needs to be addressed. If you execute the application initially at least 3 times, CA MAT Performance Management Assistant will automatically establish relevant thresholds using statistical algorithms. With CA MAT Detect plugin for Zowe CLI integrated in the pipeline, the automatically detected alert is retrieved and put it in the email to notify developers.
Let’s use a simple example. Imagine that, while changing a COBOL application DEMOCOB1 used by batch job DEMOCICD, the developer has accidentally increased a loop counter from 2000 to 12000 without noticing it. This single character typo would straight away increase the execution time of code under loop by 500%.
After committing this code change, the automatic pipeline transfers the code to the next stages; right through build, deployment and testing stage. While the testing is executed, the Performance Management Assistant component measures the job performance in the background. After the testing is complete, the pipeline script issues the CA MAT Detect command zowe pma get alert-by-job DEMOCICD to check if the performance is still within normal ranges. If the CA MAT Detect reports a performance alert, an email is automatically sent to the development team, notifying them that the performance issue needs to be addressed. In the email example below you can see a huge deviation of CPU time from the average value, which was introduced by the unintentional loop counter change.
3. Performance on Commit: THIRD STEP — ANALYSIS
Now the developer knows there is a problem, but they need to find out where to make a change, which part of the code is causing the deviation. The industry standard to figure this out is to use performance samplers. Samplers are able to measure the application and give detailed information on where each part of the application is spending time.
Zowe “CA MAT Analyze” plugin comes in play here. It provides a simple set of commands that any developer can run right from the IDE terminal to be able to measure and analyze the problem.
In case of our application DEMOCICD, which was reported as having a performance issue, the developer can use zowe mat monitor codeview moduleto find out the application module that caused the performance degradation. From the command output below we can see that the main CPU consumption occurred in the application address space program (.APPL) called DEMOCOB1.
For further analysis, another CA MAT Analyze command comes in handy: zowe mat monitor delayview. It shows the main delay types from the performance measurement. In our case, it is Program Active, which means that the running application was actively consuming the CPU resource. This correlates with our “unintentional loop” situation.
To isolate the problematic code, we have to drill down into this delay category. By passing the major and minor categories as parameters for the delayview command (in our case it’s the same category Program Active), we can retrieve the exact offset locations of DEMOCOB1 module where the top consumption occurred:
By browsing the listing for DEMOCOB1 in the IDE and taking a quick look at these offsets, we can easily determine that the code responsible for the delay is under label SUBSCRIPT2–100:
All that is left for the developer to do to complete the code analysis is to look up the place in the source code where this label was called and the performance delay - a long loop - was introduced.
SUMMARY
The beauty of CA MAT Performance on Commit solution is that two plugins are working together, so whenever CA MAT Detect reports a deviation, CA MAT Analyze has already got a sampling measurement for analysis in place, taken at the time of the deviation. The developer already has data for analysis from the time the test was run and issue occurred, so they don’t need to rerun the test and can directly jump to the analysis part described in step 3.
Check out this video to get a better understanding of how these plugins work together for your development pipeline:
This is just an example. Your workload and pipeline may look different. However, CA MAT “Performance on Commit” plugins may be used at any stage of your testing.
NEXT STEPS — TRY IT!
I encourage you to get the new release of CA MAT, version 12 and give these new CA MAT capabilities a try:
https://www.npmjs.com/package/@broadcom/mat-detect-for-zowe-cli
https://www.npmjs.com/package/@broadcom/mat-analyze-for-zowe-cli
For additional details on CA MAT Performance on Commit plugins usage visit our documentation.
Questions and comments are welcome directly at www.broadcom.com/CA-MAT-Learn-More
For additional information on how to shift left the overall process of testing batch applications, check out Sujay’s Solomon blog.
