How to Automate Unit Testing with Limited Resources using Test4z
Automated unit testing is widely accepted as a high-value, high-impact software delivery practice. It enables developers to catch defects early, significantly reducing debugging time and improving overall code quality and confidence.
But how do you automate unit testing with a large portfolio of complex, monolithic applications with few, if any, existing tests? Where do you start, especially with development capacity so limited?
The Test4z introductory article, “Unit Testing for Mainframe, with Confidence”, featured the 6 Core Principles of unit testing (i.e., the best practices). This article addresses the practical reality of unit test automation voids by offering recommendations that improve application vitality in a capacity-constrained and risk-managed way.
Challenge
For many, testing is a manual process. This approach is very costly, extremely stressful, and surprisingly common. According to Forrester’s Developer Survey 2023, 43% of testing is still done with manual practices.
Yet each manual testing cycle is effectively one-off, throw-away work that runs counter to the Agile and DevOps principles of continuous value delivery and shift-left automation.
The ideal solution is to proactively deploy a suite of unit tests to achieve a desirable level of code coverage across each application and the portfolio.
This approach, however, is impractical in the short term due to the sheer volume of source code in most mainframe shops. The upfront investment required to create tests for millions of lines of code is massive and teams simply don’t have the capacity to tackle ‘testing debt’ in a comprehensive way.
There are, however, a few practical alternatives that address the most common types of risks mainframe teams face.
Primary Risk Drivers
- Staff risk — Due to retirements, most organizations are losing in-house application expertise including knowledge of process flows, data structures, and controls, and documentation is often lacking. This lost expertise, that was built over decades and relied upon for manual testing, will not be replaced easily. For this reason, investing in test automation should be a top priority.
- Business risk — Changes to some applications present a greater risk to the business than others. This can include brand/reputational risk associated with, for example, the incorrect processing of business transactions. It also includes innovation and competition risk (i.e., delivering value to customers continuously), regulatory and compliance risk, etc. The more business risk, the greater the need for reliable test automation.
- Architecture risk — Often built as self-contained monoliths, mainframe applications carry architectural risk. For example, the risk of refactoring should be weighed against the risk of manual testing (i.e., the status quo). Unit testing application in-place mitigates the architectural risk of refactoring, replatforming, etc.
Recommendations
The following recommendations represent a pragmatic approach to building unit test coverage to address the various risk vectors.
There’s no way around it, test automation requires effort. While the AI bot cavalry may or may not be on the horizon — I’ll address this later — these recommendations provide teams with ways to demonstrate the value of unit testing with limited upfront investment. They can also help build a business case for added investment based on increased developer productivity and application agility.
- Initiate (as-you-go): Adopt a policy that stipulates code changes will not be promoted to production unless the changed section (or sections) have appropriate coverage. The developer responsible for the change should also be responsible for the creation of and passing of the test(s). This approach will gradually increase coverage for applications that require changes/fixes (i.e., self-selecting) without having a material impact on productivity. If a developer is changing code, they are already knowledgeable enough to test the relevant sections (e.g., COBOL paragraphs).
- Mitigate (record/replay): For applications that change less frequently, mitigate unit test voids with regression tests using Test4z’s native record/replay capability. This stopgap provides a simplistic way of determining if a subsequent change to the application has an impact by recording test scenarios and then replaying them as needed with automatic validations. This technique is also valuable as a set of tests that must pass as the application deploys between dev/test, pre-prod, and prod environments. Performed by a central team or a manual tester who has mastered this aspect of Test4z, these tests can also be run, for example, before/after a compiler or middleware (e.g. CICS) version upgrade as an early warning system.
- Triage (targeted build-out): While the first recommendation improves code coverage for applications as they’re updated, most businesses would benefit from creating unit tests proactively for the most critical code. Partner with product owners to determine which applications would benefit most from unit test automation (i.e., faster changes, higher confidence). Those applications with significant business risk and relatively frequent updates should be the focus here (i.e., the low-hanging fruit). Assign a developer who understands the application or a developer who will maintain the code going forward (creating unit tests can be a valuable learning exercise).
Why Not Wait for AI to Write the Tests?
It might be tempting to wait for GenAI to assist with test creation but, as described in my colleague Venkat’s recent article, “Tackling Today’s Mainframe Challenges with GenAI: A Pragmatic Approach”, GenAI needs training data to learn. How are you going to train the code generators?
By building out a unit test portfolio over time using the techniques described above, you are preparing a training data set that will deliver a handsome ROI as code generators mature. Then down the road, in true risk-managed fashion, you can apply the AI-created unit tests to lower-risk applications/sections rather than on the code with the significant business risk (i.e., don’t test an autonomous vehicle in mid-town Manhattan).
Conclusion
The need for test automation in mainframe business applications is clear. While test automation requires effort, it does not and should not be an all-or-nothing venture. There’s no need to boil the ocean. Remember, manual testing is still dominant throughout IT and Test Driven Development (TDD) is more aspirational than reality. We have room to change the culture of mainframe application development thoughtfully and purposefully by keeping our eyes on the overall goal: greater code change confidence and application vitality.
Test4z is part of the Mainframe DevOps Suite so there is no procurement barrier to unit test adoption. If you’re licensed for the Suite (or Brightside) and would like to explore Test4z, visit the website. Your new-to-mainframe talent might already be familiar with tools like Test4z, which is similar to distributed unit testing tools like JUnit, TestNG, and PyTest, as well as the 6 Core Principles of unit testing.
Pairing an experienced programmer with a new one tasked with writing unit tests is an excellent way to grow COBOL expertise and application knowledge at the same time. This knowledge transfer investment will help ensure your code base remains vital for the next generation.
So, as a community, let’s be smart about how we apply unit testing by mitigating the risk of application changes over time. Let’s not shy away from the challenge but, rather, embrace it as a means to better support our product owners and business leaders.
Remember, ‘A journey of 1,000 miles begins with a single step’.
Read more at the Modern Mainframe.
