Principles of a Test Approach

It’s quite typical for large organisations to have big unwieldy Test Strategy documents, detailing who does what, when and where. The reality is, these documents are mostly produced at the beginning of the project, and once they’re approved no one looks at them again.

When I come across Test Strategy documents, they typically have the following characteristics:

  • Too long
  • Growth over time has lead to contradiction
  • Overly prescriptive
  • Not aligned to working practices
  • Adherence/compliance is very low

I often see things like “teams should work in an Agile way, with a two week Sprint and aim for XXX% automation”. Perhaps a slight over exaggeration, but you see the point.

I prefer to work with Test Approaches instead of Test Strategies, simply because I find it gives a framework within which teams can be creative. A Test Approach provides principles instead of extensive prescription of working practices. Teams are free to work within the principles, and on larger programmes, perhaps a given tool kit.

I like to split these principles into two distinct areas; Testing Principles and Checking Principles. If you’re not sure of the difference between the two, then James Bach offers the following distinction:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modelling, observation, inference, etc.
Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

Testing Principles:

These are roughly inspired by some of the ideas presented in the Government Digital Service blog article “Agile Testing Manifesto”.

Inference through observation over abstract system interaction
Exploration over acceptance
Testing supported by Checking over high level automated Checks
Collaborative ownership over detached objectivity
Defect prevention over defect reporting
Outcomes over documentation
Context over standards (process, repeating the past?)

Checking Principles:

White-box Checks over Black-box Checks
Early execution over late confirmation
Interfaces over user journeys
Atomic Checks over Linear Checks


Testing Principles

Inference through observation over abstract system interaction

This principle is all about de-valuing the popular approach in the industry today of testing through high-level application interfaces, especially where business logic is abstracted away from the UI. Many organisations are moving towards micro services. Which means that much of our logic happens well away from the browser.

I’m not saying that testing the UI isn’t relevant, I’m saying here that checking only through the UI is a bad idea, where the complexities of your system are abstracted away. Working with REST APIs, Message Brokers, Event Stores, View Stores, logs, metrics, health checks, monitoring and alerting all give us important interfaces where it’s usually more appropriate to check than in the UI. Context is king here!

Exploration over acceptance

This again alludes to reducing the prevalence of black-box system testing through UIs, valuing exploration of products to understand what they can do instead of focusing on simple acceptance of a product which can only tell us that it does what it should do. This principle seeks to drive information gathering and to build practices that will uncover deep, insightful understanding of a product, and hopefully uncover some great bugs!

Testing supported by Checking over high level automated Checks

From the definitions of Testing vs Checking above, Testing almost certainly cannot be automated (not right now anyway).

There is no suggestion with this principle that automated checks hold no value. The suggestion is to acknowledge and understand tools and their limitations. I believe that all of the automated checks in a system should reflect, as much as possible, all of the knowledge about the system. Testing as a craft aims to help us acquire knowledge. The results of performing some testing, usually drives some implementation, update or removal of some automated checks as our knowledge changes and evolves.

We also use automated tools when testing. Sometimes that might be monitoring systems, log aggregation or an automated check suite that allows us to speed up state transitions in the system.

Our goals however are to provide as much information to stakeholders as possible, and therefore we should value testing supported by checking over a blind over-dependence on checking.

Collaborative ownership over detached objectivity

We’ve kept this one entirely unchanged from the initial GDS proposal. Here’s what Neil Fletcher originally proposed:

“When an external team tests your product, it has the benefit of being able to take an outside view. However external testers can’t quickly incorporate usability feedback into your business model like your own team can. External testers won’t be aware of the reasoning that takes place during development, nor will they have insight into the high-risk areas of the service, which consequently need the most testing attention.

To build a quality product, testing activities must be fully integrated with the development cycle. The same team that creates the product must do the testing. This ensures the whole development team owns the quality of the product, rather than one individual or role.”

Proactive bug prevention over reactive bug detection

Moving from reactive to proactive practices is a good way to improve efficiency and to often reduce costs. When it comes to bugs, it’s well known that fixing an “escaped” defect that makes it into production is more expensive than finding it earlier. This principle goes a little further however, pushing practices that are designed to reduce the amount of issues before they are coded.

Things that we use to do this stem from good agile practices; stand ups, retrospectives, refinement and grooming. Some are more specific like “kick offs” or “three amigos” before building and statically testing designs and prototypes.

This principle doesn’t de-value testing or detecting bugs. It’s about driving testing earlier to improve quality through out our delivery process, it does not save time on testing later but it can help to reduce rework.

Outcomes over documentation

This principles lends its origins to broad agile principles around documentation. To be clear from the start, there’s no suggesting that documentation isn’t needed, but instead we should be looking to produce just enough of what’s valuable when it’s valuable.

If I had a choice over writing a manual test script or performing some testing, I’d go for performing testing every time. This principle is especially prevalent in an industry where documentation is traditionally heavy.

We look to use documentation to record outcomes instead of documenting desired outcomes. This might be an aggregation of metrics or logs, performance results or an API contract. There’s also a place for heuristics, mind maps, oracles and charters.

The key is that documentation should enable outcomes, not prescribe process.

Context over standards (process, repeating the past?)

I’ve struggled with the wording here for a few months now. I can’t decide if it should be “standard”, “process” or “repeating the past”.

ISO 29119 and IEEE 829 have made some attempts to standardise an approach to testing, as have qualifications like ISEB/ISTQB. These broad bodies of work lack a framework for the application of context, something that’s extremely important to ensure we focus specialist attention instead of general high level cover when we test.

When some text changes, do we need to do a full performance test? A penetration test? Probably not. We may also choose just to test this change with users more extensively than we would with a change from MySQL to Postgres, with research needed around the text and accessibility needs. We may choose not to carry out much exploratory testing for this change.


Checking Principles

White-box Checks over Black-box Checks

Traditionally the automation pyramid has helped to drive this principle.

White-box tests support one of our Testing Principles; “Inference through observation over abstract system interaction”. It does so because White-box checks allow us to make finer grain observations of lower level behaviours in an application.

By favouring a White-box approach to checking we give ourselves more opportunity to make accurate inferences from our observations.

There’s still a place for Black-box checks of course. Most consumers see systems as a Black-box and we should ensure that their perspective of the system works, but we gain opportunities for better observations when we favour White-box checking, and the automation pyramid helps with this.

Early execution over late confirmation

This isn’t a new concept. We have automated checks, we run them as soon as possible against as small a change set as is possible and run them as often as possible.

Automated checks should provide two things; an expression of our current knowledge of the system and a safety net for change. If they randomly fail, take a long time to run or are maintained only by a subset of the team, then they are not serving us properly and this problem must be addressed.

Coupling automated checks to the code base that has the features the checks check is essential here. If a team member has to checkout another code base then it’s very possible for dysfunctional behaviours to start to appear. For example, testers could be made “responsible” for the test code bases and the developers “responsible” for the features code bases. This is a bad thing because this is start of silos, which leads to bottlenecks and ultimately, slower delivery.

Interfaces over user journeys

There are many interfaces to interact with a modern web system, and most of them are commonly either not considered or not given the attention that they deserve. A non exhaustive list includes:

  • Logs
  • Metrics
  • Health checks
  • Heart beats
  • HTTP
  • HTTPS
  • Message Brokers
  • Persistence

The most tested interface is usually over HTTP(S), the public consumption point of web services. The rest are quite often left by the way side, passively tested by treating a web service as a black-box.

In an Service Orientated Architecture, presentation tiers are decoupled from back end services. The integration point is often not the implementation but the contract, with the theory being that anything could implement the contract.

This means that the user journey through a user interface often can’t “prove” that the underlying services work. It can passively prove physical integration with back end services, but can’t guarantee that contracts produced and consumed by both systems are correct (although it does give superficial confirmation that a user can take a journey from A to B and infer without deeper observation that something appears to have happened).

Testing at each of the systems interfaces and supporting this testing with robust, fast checks improves our capability to reason about a system, and also helps to ensure that we fulfil all of our users needs. It also allows teams to build up a picture of quality as they add more services into the mix.

So user journeys checks are valuable, they just can’t give us the feedback that they are commonly thought to give. We must understand the tools that we are using and the information that they are capable of giving us.

Atomic Checks over Linear Checks

Automated checks should be able to run entirely on their own, in isolation. They should not depend on one another. If the system needs specific state as a precursor to the check, then the checks setup code should be responsible for this.

Linear checks should be avoided at all costs; any check suite that executes checks that are dependant on application state being achieved by a check that has ran in the past are brittle and will inevitably lead to a de-valued suite as trust is reduced every time they fail.


In my experience, Test Approaches based on principles instead of Test Strategies focused around prescriptive practices create more opportunities for testing, promote creativity and increase engagement.