Testing beyond requirements

I often hear people talking about testing as validating and verifying requirements. But testing is so much more than that.

When we focus on verifying and validating we will most likely only find and look for those things we seek. It’s embedded in the meaning of the words themselves.

According to the Oxford Dictionaries;

Verifying is to: Make sure or demonstrate that (something) is true, accurate, or justified.

And validating means to: Check or prove the validity or accuracy of.

With this mindset we tend to focus on confirming whether the function or product works as stated. But we forget to observe what else is happening . It’s like reading your horoscope that says you will meet a blond stranger — and suddenly you are only paying attention to blond strangers.

Testing is to learn about the product through exploring and experimentation but when we focus on verification it turns into a demonstration of the product:

“Yes, I can login to the website.” “Yes, I can add some products in my shopping cart.” Does this seem familiar? However the demonstration only shows that the product can work, once, in some kind of controlled circumstance. It’s the process of verifying which leads us to think in this way.

When our brain is targeted towards verification we tend to forget to ask questions like:

What if…? What else? What happens when…? Who is this for? Why is it designed like this? When can this be used? How can this function be used differently then expected? What value does it provide? To whom?

By challenging the product or the system you are testing you will discover a lot of new things. When moving away from your confirmation bias you will increase the opportunity to find the bugs/defects you don’t want your users to find. In worst case those bugs which will cost your company a lot of money.

But we need to make sure we meet the requirements!

The most common argument I get when advocating for testing over validation and verification is “but we must make sure the requirements are fulfilled”.

Testing requires multiple information sources. The written requirements are only one input to the test analysis to decide what needs to be tested. If we want better products and software we must go beyond checking. Since we can’t test everything we must know what is the most important to test. There are many parameters which affects that decision and this is yet another skill which you need to have when testing. This is where I see many focus on a too narrow approach choosing verification over testing. What you can do instead is to focus on risk. What is the worst thing that can happen? What if we can’t proceed to checkout? Instead of checking a few specific flows — be creative and explore these flows from different perspectives.

We have both explicit and implicit requirements and we can’t write down every aspect of a requirement. Some things must be tried out before you understand it. Some things are considered so obvious that it is just not documented. There is always a possibility for interpretation and ambiguity in requirements whether they are written or spoken.

So why don’t we spend more time making sure the requirements are perfect? Perfect requirements is an illusion. Even if it would be possible to create perfect requirements we would be out of business before we got our product to our customers. The quicker we can challenge the product by interacting with the software or design, the faster our feedback loop becomes.

By challenging the product from different perspectives like different quality characteristics , different users, different context and applying different test techniques the more likely we are to find those critical bugs which in worst case can put us out of business.

So let’s step out of the verification mode and start testing!

If you are to dive into testing beyond the requirements here is a bit of reading which I can recommend to start with:

Rickard Edgren http://www.thetesteye.com/papers/TheLittleBlackBookOnTestDesign.pdf

James Bach
Testing vs Checking http://www.satisfice.com/blog/archives/856

Michael Bolton

Anne-Marie Charrett