The hard thing about testing
Why are bugs in software inevitable? One of my arguments is that knowing what to test is one of the most difficult parts of testing itself. Bugs creep up in the areas we never thought to look. As software gets more complex, those undiscovered areas get bigger and more disparate.
I recently had the privilege of speaking about this topic at the QA Summit for Digital Marketing. I wanted to summarize this particular argument here, alongside a few of the slides from my session.
You can listen to the entire session and view the slides here.
It’s easiest to see the loopholes in our test plans by looking at an ostensibly obvious solution to a problem. So, let’s get away from software for a moment altogether.
A lightbulb moment
Suppose we are aliens from a far-away planet and we’ve finally decided to land our UFO (for us aliens, it refers to an Ultra-Fast Object) onto planet Earth. As all UFOs do, we land near some remote farmhouse in a cornfield in Iowa. We exit our aircraft, quietly head into the barn, and come upon a simple invention that we’ve never quite seen before: The lightbulb and switch.
We observe how the light bulb works in relation to the switch. What are the behaviors we notice? Flip the switch on; the lightbulb glows. Flip the switch off; the lightbulb shuts off. We can screw in the lightbulb and take the lightbulb out. Through centuries of advancement in our own home planet — you know…full of robotic maids, self-cleaning homes, and invisible paint — it never dawned on us to create a switchable light source!
We take copious notes on this novel idea before escaping back to our home planet. The next morning, after a generous helping of alien pancakes and coffee, we get started on replicating this great find. (At this point, let’s just assume we’ve got all the other necessary tools, like electricity, screws, and filaments, to build the components of our alien light bulb and switch. We’ve got robotic maids after all!)
After a few hours, we’ve come up with a perfect replica — we think. Here’s how we confirm that our lightbulb and switch works just like the one we saw outside of Sioux City the day earlier:
At this point, we’ve solved it! Our lightbulb and switch work as it was intended to! Or, has it? Are there scenarios we may be missing?
- For instance, what happens when we unscrew the lightbulb while the switch is turned on? Based on our earthly observations, we’d expect the lightbulb to turn off, but, what if it didn’t? Did our version of the lightbulb happen to somehow store the energy needed to keep it lit even without being tied to the switch? (The argument about whether this would technically be a feature or bug we can leave for another day).
- What happens if we turn the switch on and then screw the lightbulb in? Would the lightbulb immediately turn on as the earthlings would expect? And, if it didn’t? Perhaps, we’d find that the order-of-operations matters in our version — that the bulb needs to be set prior to the switch setting. There’s another scenario that may have escaped us.
- And finally, what if we took the lightbulb out altogether? The lightbulb shouldn’t be affected by toggling the switch at that point…right? But, what if the untethered lightbulb lit when we turned the switch on? Maybe we unintentionally built wireless electricity!
Here’s a more rigorous set of tests to perform:
The ridiculous isn’t so ridiculous
While you may think that some of these potential what-ifs sound ludicrous, they could fail. And if they do, there is a fathomable explanation as to why a seemingly obvious result doesn’t actually happen in the way we expect.
In software, these at-first-seemingly-ridiculous cases happen so much more frequently. A flawed conditional, a mis-scoped variable, an uncaught exception — there are too many little mistakes we make in software that can lead to very odd behavior. As developers, we need to get out of the tunnel-vision mindset of testing (the on/off scenario), and start uncovering the “unlikely” corner cases that happen quite often in our work.
