Why quality matters
by Simon Tomes, Quality Assurance Lead at Ostmodern
This is a blog version of a talk given at Ostmodern’s company-wide meeting on 31 August 2018.
Bug magnet for life?
During my career as a tester I’ve had a reputation as being a bit of a bug magnet. And this creates an uneasy feeling. Does that mean I improve quality? Am I hindering quality in some way? What part do I play in building and releasing products?
I used to get a kick out of finding bugs and making the final call on whether we should go live or not. It’s a bit of a strange dilemma though — a team would be pushing to release and I’d be the only person saying we shouldn’t. It was a little lonely and I’d become a quality gatekeeper.
More recently in my career I have learnt that being a quality gatekeeper is not good for a team. Who was I, as an individual, to say if something was the right level of quality such that it was good enough to go live?
Instead, I have learnt to collaborate more with the teams I worked with, actively seeking ways for the whole delivery team to own quality and make collective decisions. I had to question my view on quality and testing to do this.
No longer was I a quality gatekeeper and no longer was I a bug magnet. I – with the help of many others – could be way more than that!
Find a level of quality that matters
Quality is a funny thing. You can’t really grab quality, it’s not so tangible. There is a lot of useful information that acts as a proxy to quality and you can define quality criteria. My team (of QA engineers) happens to have the word ‘quality’ in its title. However, all of us across the company contribute to quality in many ways.
I have a feeling we all know why quality matters. Yet quality is a complex subject.
Across the globe, across many different industries, we see tweets that pull products apart and destroy a company’s reputation. We’ve seen app store reviews that turn us off downloading and installing. We’ve read the stories about data breaches. We know the impact of this.
More than ever, some important level of quality is expected. But it’s all subjective. Well, except for whatever’s written in a contract — but that’s another conversation.
With all these examples in the world, quality is still hard to define. Like trying to describe the colour blue or an emotion such as happiness — there are multiple ways to approach these. What matters is our awareness of quality.
With too high a level of quality, you might deliver a product late focusing on something people don’t care about. Too low and people will lose respect for you, and that particular quality level interferes with providing people with value.
It’s about finding a level of quality that matters, and we do that with quality awareness.
Ask questions and discover useful information
How is quality related to testing? Testing is often an exercise in asking questions, such as:
- What happens if?
- Why did that happen?
- Where should this work?
- How many times does this happen?
- Who uses this?
- When should this happen?
Every time we ask a question, we are contributing to quality in some way. We discover information. And that information is (hopefully) useful.
So, another way of thinking about quality is as “information”:
- What information do we need to give us confidence that we’re building in quality?
- What information should we discover to identify things that threaten the value of our delivery?
- And then, what information should we discover that helps defend that value?
A Quality Awareness Strategy Model
Here’s a Quality Awareness Strategy Model (QASM) that helps us talk about quality and testing, inspired by Dan Ashby’s Model developed at eBay.
Start at the top and you’ll notice there’s no ‘one size fits all’. Context is important — contract details, team setup, client experience, product goals. And a particularly useful part of context are risks to drive test ideas.
Prior to my realisation that testing is information gathering, I used to think of testing as something that we do to break software. I was a hammer. But testing isn’t about breaking anything, it’s about discovering what’s already broken and uncovering plenty more useful information along the way. This informs our decisions about what to work on.
Instead, think of testing as a torch. Take a look around your office and imagine you had to test it — how would you do that? You could use risks to guide your testing. There are tons of risks worth exploring. There are security risks, usability risks, accessibility risks, performance risks etc.
You and the client need to work out where to shine the torch. Time always seems to be against us, so I encourage you to use risks to prioritise what information you’d like to discover.
Two types of testing for two types of information
The information you use and information you discover typically come in two forms related to testing approach:
- Assertive Testing, where the input is explicit using Given/When/Then scenarios to check against manually. The output tends to create pass and fail rates and leads to bug discovery and tracking.
- Investigative Testing, where the input is tacit, implicit or unknown. Test charters are defined and time-boxed, and unscripted notes are documented per session. The output tends to amplify problems, questions, ideas and praise.
Think of testing as though it is like evidence-based journalism — news sites such as Reuters or WikiTribune. Imagine you’re a journalist turning up at a breaking story. There’s action you expected to happen, such as ‘the Prime Minister will meet this person and give this briefing’. It’s a script and set of scenarios you check against — Assertive.
These scenarios you heard about were clear and concise and you’ve checked for the existence of the fact. Now, unexpectedly, a bunch of protestors arrive. Do you ignore them? Certainly not! This is something you’re keen to report on and it’s fine that you don’t know what to expect.
You capture notes on your observations and ask questions to learn more — Investigative. You document assertive and investigative information and add it to your compelling story.
Always be testing
Information feeds confidence indicators and another round of test ideas. How do you as a team collectively understand confidence? Working individually can sometimes make it harder to discover and share useful information.
The Quality Awareness Strategy Model encourages people to pair more often in different testing related activities. In essence, a default to “Always Be Testing”.
For example, a developer might pair with a QA engineer just before they commit their code. They could tell a story about how they developed the thing they’ve just worked on, perhaps opening up to some potential problems and code they had to refactor in the process.
Or maybe a QA person debriefs their testing session with a product owner, developer and designer, and shares how they tested as well as what was tested.
Collaborative conversations amplify quality awareness and this helps a team learn if its product is moving in the right direction. Testing becomes a learning exercise for the whole team to discover more about the product they deliver.
We can test anything. An idea, process, paper prototype, full blown hi-fidelity mockup, api, database, data model diagram, stories, working app, etc.
Information from testing activities gives us a level of confidence in ourselves and our ability to deliver products we can feel proud of. This information provides our clients with confidence in our service.
A word of warning about information
We should be aware of our cognitive biases: there are said to be about 188 of them. I’m particularly wary of our tendency to get excited by metrics, such as the count of automated tests going up, and bug counts going down.
Do those trends mean quality is moving in the right direction? Perhaps it’s a loose proxy. Those types of metrics are information that contribute to our awareness of quality — but it’s not quality in of itself.
Session-based testing notes, showcases with our clients and real user feedback are some examples of good places to learn about the quality of our deliveries.
Challenge the industry norm
Let me give you one final idea to consider. One of our strategic themes as a company is about taking the initiative to be innovative. I think we have an opportunity to move the agency industry in a different direction, away from contracts that have P1 and P2 bug counts written into them.
There are alternative ways to this industry standard which I believe are worth exploring. But that’s another topic for later. In the meantime, always be testing.
One challenge across the whole testing industry is to talk about testing and quality in a way that resonates with anyone who is part of a business that builds and deploys products.
I hope this post pushes forward an understanding of quality and the role of testing in this. Let’s learn from each other and find an opportunity to collaborate.
Thank you for reading!
If you like this article, give us 50 claps. If you really like it, let us know why. Leave us a comment with any question or opinion you might have on the subject.
You can reach us on Twitter @Ostmodern to continue this and other discussions.
Follow our publication for more stories on tech, design, video and other interesting topics.
