Software Testing Principles

I’ve been working with Luke Liu helping him define what the “testing mindset” is. Along the way we took a look at various lists of principles different people have put together.

In particular, we looked at ISTQB’s seven principles of software testing, The Seven Basic Principles of the Context-Driven School, and Alan Page’s Modern Testing Principles.

It was an interesting exercise to analyse, discuss, deconstruct, and reconstruct these into principles that resonated with us. We saw similarities and contrasts. I’ll post the principles as we found them published without comment (although I have a LOT of comments on many of them) and then post what we came up with references to the original principle that inspired it.

ISTQB Principles

1. Testing shows the presence of defects, not their absence.
2. Exhaustive testing is not possible.
3. Early testing saves time and money.
4. Defects cluster together.
5. Beware of the pesticide paradox.
6. Testing is context dependent.
7. Absence-of-errors is a fallacy.

Modern Testing Principles

1. Our priority is improving the business.
2. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
3. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
4. We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.
5. We believe that the customer is the only one capable to judge and evaluate the quality of our product.
6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
7. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

The Seven Basic Principles of the Context-Driven School

1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

After looking at these, synthesising them, and adding our own beliefs and experiences, we have these as a set of beliefs. Where we cite an existing princple as an inspiration or as a response, we will use the code CDT for Context Driven Testing, ISTQB, and MT for Modern Testing.

Aaron and Luke’s Testing Beliefs:

1. Testing is information gathering and reporting
2. Test Coverage is always relative to some model (ISTQB 2)
3. What we test, and how much we test, should be based on models of risk (MT 7)
4. It is often better to perform a variety of tests than repeat tests (The minefield heuristic: https://www.satisfice.com/reasons-to-repeat-tests) (ISTQB 5)
5. We can’t ever be certain there are no problems (ISTQB 1)
6. A product can meet all its stated requirements, and still be useless. A product can fail to meet all its stated requirements and still be useful. (CDT 5; MT 5)
7. Testing can happen at any stage of any SDLC, and it is usually better to begin testing earlier, rather than later (ISTQB 3)
8. Shorter periods of testing that happen more often are better than longer periods of testing that happen less often.
9. There are no best practices, only good practices in context. (CDT 2)
10. Change happens, and that’s a GOOD thing. Therefore, our practices should be as responsive to change as possible. (CDT 4, CDT 7, MT2)
11. Testing is a performance, not a procedure, and certainly not an artifact. (CDT 6, CDT 7) (See also: Test Cases Are Not Testing)
12. I value healthy teams of people building useful products for people consistent with my ethical beliefs in a way that minimises waste. My focus is on reporting threats to that value, and the values of my stakeholders. (MT 1)

This is a draft; we’d be keen to hear what you think.