Five-Fold Testing System — #5: Evaluation
“An oracle is a means by which we recognize a problem when we encounter one in testing.” — Michael Bolton
As testers, we test. When we find things that seem to be problems, we report them. However, not everything that we report ends up as an agreed-upon problem. This final article is about the various things that go into how testers report their findings/results. How effectively a tester can communicate their findings is crucial in forming the perception of that tester’s usefulness to a given team. If they can provide results that are pertinent (relevant!), interesting (helpful!) and comprehensible (simple!) then that tester is well on their way to rock-stardom, and their future reports will be given much more weight.
I once saw a man in downtown New York City who most would describe as ‘crazy’. He wore long hair, was unkempt and unshaven, with suitably raggedy clothes. He walked against an awesome flow of oncoming human traffic in the middle of the street, and as he walked, he repeatedly yelled out variations of the following phrases, “I see dead people! I’m seeing a lot of dead people right now!” etc., etc. I noticed from his hand gestures that he seemed to be referring to the very people around him who were marching along in their throngs to the beat of the city.
The memory stayed with me for many years, firstly because I felt I understood what he was getting at, but more pertinently here, because I felt that in some ways, this man exemplified (for better and worse) an interesting model of software testers. Allow me to explain…
1. The Experience Oracle
That man on the city street in New York, highly-sensitive and instinctively attuned to man’s higher purpose (or so I like to imagine!) was clearly perturbed by what he saw; perhaps to him this everyday city-scene looked like a sea of mindless, soulless automatons rushing around life, working pointless jobs accomplishing next to nothing of ultimate importance. As such, he blurted out that all he was seeing was “dead people”. Now, at this juncture, all he was actually doing was noticing something based on his own gut-feeling (a feeling which itself is based on his own models and ideals, i.e. limited to him).
His feelings are a valid oracle (a means by which he can recognize a problem), however, one’s personal feelings are not exactly the strongest of oracles.
When you are tempted to think “this is a bug because it’s obviously a bug” consider re-framing that to, “this is a bug because I observe that the product behaves in a way that violates requirements X, Y, and Z, and those requirements are valued by my clients.” — Bach, Caner & Pettichord, Lessons Learned in Software Testing
2. The Inference Oracle
The first (and perhaps least visible) inference one can make to strengthen that oracle is to ones own past experiences or knowledge. From these, we can make inferences that have a little more impartiality, backstory, and ultimately, backbone. If what you have observed is in contradiction or inconsistent with those inferences, then you can more wholly substantiate a claim that this observation is, in fact, a problem. That man, for example, may have been raised in a rural part of the country where strangers smiled at one another and said good morning, where time moved slowly, and where one wasn’t sequestered away in a dark cubicle all day. Likewise, he might merely have once read a book where a more traditional, rural behaviour was described in a favourable light. In each case, he now has something else to reference on top of his own gut feelings.
“The first observations we make are usually tacit, and are in need of being made explicit.”
03. The Conference Oracle
Going down one square from the diagrams initial starting point of ‘experience,’ we get to the idea of Conference, another way to reinforce and strengthen a test oracle. During an exploratory testing session, I may notice a strange layout/spacing issue on one of our dealer websites, or even a basic typo. I know these to be strange based solely on my exposure and life-experience with web applications (both as a tester and a user). After raising these with the developer or product owner, my concerns may ultimately be dismissed (“that spacing issue is Design’s problem, nothing we can do about it!” or, “that typo comes from the third-party, they’re the ones who’ll need to fix it!”). There’s nothing wrong with these responses of course, in fact, this is fairly common occurrence. The key phrase there was, ‘after raising this with…’.
The first observations we make are usually tacit, and are in need of being made explicit, or at least, more explicit than before. Comparing observed behaviour to our own inferences might be fine, however, one of the first and most common responses is simply to bring someone else over to your machine and get them to see what you are seeing (and then pray they don’t say “it works fine on my machine,” five minutes later!). As the diagram insinuates, we then go from (personal) ‘Experience,’ to (mutual) ‘Conference,’ and we can — at the very minimum — agree that this constitutes at least one more layer of scrutiny than we had before.
04. The Reference Oracle
Finally, we know that in order to give our claim an even further layer of legitimacy, we can supplement it with shared references. If I find an issue with a story that has clear acceptance criteria which are being broken, I can reference them. Likewise, if that man in the city were able to cite something from popular culture, a phrase or sentiment that many people knew and related to, something which was effectively saying the same thing he was saying, then he may have found that his observations would have been taken more seriously (and perhaps would not have been so easily written off as ‘crazy’).
Take aways
- It takes all kinds of people to make a world!
- Testers are a bunch of nuts.
- But we’re on your side!
- A testers reputation hinges on their reports, and their reports can be strong, mild, or pretty tame.
- The analyses of oracles given above could really all just be replaced with one basic imperative: be thorough!
