Testing for compliance — or quality?
The world is full of software that perfectly meets all documented requirements but offers no added value to its users.
When I — a long time ago — studied software engineering, quality meant conformance, i.e., conformity to requirements. I also learned that quality assurance meant all actions taken to enable, produce, and verify quality and that testing was an essential quality assurance method.
Today, Wikipedia defines testing as the study of the artifacts and behavior of the tested software through validation and verification. Testing is defined as an activity, leaving the purpose open.
The definitions I learned in computer science school match the Wikipedia definition of compliance testing: Tests that determine whether a process, product, or service meets the requirements of a specification, technical standard, contract, or regulation. Oddly enough, these definitions are yet to include the word quality.
What is quality?
Nowadays, I avoid the temptation to indirectly define quality by four dimensions: Benefits, Functions, Processes, and Experience. Let us think of something everyone knows, say Paypal. People use it to transfer money online; that’s its function. They are looking for simplicity, security, and low-cost benefits. Using PayPal involves various user processes that the software guides the user through. The user’s emotions when using PayPal, such as joy, frustration, or success, make up the experience. Whatever quality is, it can hardly exist without all these four dimensions.
Testing usually focuses on meeting functional requirements, that is, making sure that the software does what is specified in the conditions and does nothing else. There may also be non-functional requirements, such as response times, the number of transactions processed in parallel, and the security of user data. Standards and regulations, such as SOC2 or GDPR, bring additional requirements, as do Paypal’s agreements with credit card companies. While these have little to do with the actual features and value of the software, complying with them is a prerequisite for entering the business.
The ideal world
Good testers use a conformance-based view that meets requirements and an outcome-based outlook that focuses on realizing benefits. In an ideal world, meeting documented requirements would mean the user benefits, but the world could be better. Even if the software were perfect, the users are not. The realization of benefits depends primarily on the user's actions, and the user’s activities depend on how the software interacts with and guides the user. This, in turn, affects the quality of the overall experience. These software characteristics are hard to capture in compliance requirements or regulatory test cases. Again, in an ideal world, all these challenges would be solved early enough through user-centered design — but the world is far from ideal, and a tester must take on the role of the actual user.
Compliance testing vs. outcome-based testing
Compliance testing and outcome-based testing require different mindsets. Compliance testing is like setting traps: You read the requirements, imagine what could go wrong, and design test conditions under which the software is likely to fail. Outcome-based testing is more like an act of curiosity: you understand what the user needs to accomplish, imagine all the things they could do, figure out how the software should respond, and then try it out.
Compliance is a prerequisite for quality. In many industries, compliance with formal regulations is a requirement for business. You can also mitigate liability risks by providing documented proof that compliance has been adequately tested. Compliance testing helps assess the risk of failure (of a feature, functionality, system, etc.), and results-based testing increases the likelihood of success.
