Exploration vs automation in software testing
A simple google search of the term “automation vs manual testing” returns thousands of results, diving deep into this seemingly endless battle between the virtues and perils of machine checking vs human exploration in a software testing context.
After joining Devgurus I was tasked with establishing processes that would help the company increase the quality of its products. In my efforts I wondered to what extent I should rely on automated solutions and my research dug up a lot of very promising tools and techniques, some of which really tickled my developer side. As I explored the world of test automation I came to some conclusions that I would like to share in this article.
The automation dream
Understandably testing automation is a very attractive notion to both technical and non technical people. The prospect of not having to regularly perform repetitive and time consuming testing sure sounds good. The prospect of not having to pay for it sounds even better!
Conceive a test once, write it in some language you are already familiar with and then you never have to test that thing again! Boy that sure sounds good!
When something sounds too good to be true… it usually is. I don’t want to come across as saying that test automation has no role or use, it absolutely has and many. But automation is not here to replace human testing, at least not until true AI comes along, and then we will probably have bigger problems than software quality… like battling our machine overlords for example.
I strongly believe that, in reality, testing cannot be automated. This might seem obvious to some people but believe me, there are some overblown expectations out there when it comes to automation in testing. In order to understand why automation can never fully replace testers we need to, first and foremost, define a few concepts and explore what testers really do.
The testing bedrock
The title of this article is a little misleading... I plead guilty to that much. Generally people speak about manual or exploratory and automated testing, as if they were different forms of testing. Until not too long ago, I counted myself among them. This is a misconception and a very common one. There’s only testing. A part of a test might include the use of automated tools in order to assist testers in their efforts, but that constitutes only a part of the effort.
When we speak about automated testing tools we generally mean tools that periodically perform checks in order to verify that predicted outputs remain unchanged over time. So right of the bat, just by defining it, we see that this type of checks can only cover a narrow portion of what a tester does. They can only check that the application is outputting predicted values, as it did in the past. There are a plethora of tests a human would perform that these types of checks simply cannot.
This is why the closer we are to the code context the more useful automated checks become, but as we move out of that domain and eventually reach the user’s context they become less and less useful as tools for delivering quality.
Clearly we can see that humans and machines are both useful and irreplaceable in the testing process, software needs to be tested by a person and that person can (and should) rely on some automated checks as part of the that effort. We can quickly conclude that there’s no real conflict here… or can we?
Reckless automation
As Jonathan Kohl points out there used to be a lot of confusion about the extend that automation could replace a tester, simply because the prospect of fully automated testing solutions sounds so tempting. The thought being that maybe all we need is a few experienced testers designing and maintaining our automation suites to completely do away with hands on human testing. In recent times i think this issue has been getting better and most software experts understand the limits of automation but we should be careful to not let this tempting concept creep back into our minds.
Programming is about building tools that mediate human processes that aim to solve human problems, therefore is an intrinsically human activity and attempts to fully automate it are doomed because of that. Some programming has been automated though, allowing programmers to code at a higher level of abstraction, automation is, in that context, assisting humans and not replacing them. Software testing is no different and also is and should always be considered an intrinsically human activity since its there to verify all the things humans care about when using software.
I think part of the problem is that testing is often considered a trivial task, and this, I believe, is still going on today. Time consuming, sure, but trivial still, and software people are used to automating trivial tasks. To be honest I used to think this way when I was a dedicated developer. This is a huge misconception that is bound to generate bad software testing if believed.
Good testing is a complex intellectual activity. It involves creativity, logic, common sense, intuition and deep knowledge of the product. Having experience with coding is a huge help as well, not only when writing tools to help with testing but also when predicting failures. Even if you give your testers perfectly well defined test cases to work with, they cannot help but do all of the above when executing them… makes you wonder if you needed the test cases at all… uh? Maybe our time would be better spent teaching testers how to test effectively.
When we realize what testers truly do, it becomes clear that it cannot be automated.
Testing is surely not a trivial task and writing effective automated checks is a difficult skill. Writing tests comes with all the problems that writing any piece of code comes with. Bugs, poor design, maintenance, etc… If they are not careful inexperienced testers can find themselves spending all their time maintaining brittle and ineffective automated test suites while at the same time not performing good testing on the product. Automated tests should work for testers and not the other way around.
Gather ‘round folks, it’s miracle time
The other problem of thinking this way, is that it opens us up to all sort of, what I now believe are, snake oil salesmen, that will praise the usefulness and effectiveness of some automation tool to no end. All that will sound great until we realize that, not only they come attached to a pretty steep price but using and maintaining those tools is more generally more costly than manually doing the work the tools is supposed to automate and all that time is drained from the time proper testing could’ve enjoyed. Using unnecessary tools will lower the quality of the product.
Test automation cannot reproduce the thinking that testers do when they conceive of tests, control tests, modify tests, and observe and evaluate the product. Test automation cannot perform sapient testing. Therefore, automation of testing does NOT mean automation of the service provided by the software tester.
James Bach — What is Test Automation?
Conclusion
In conclusion, software testing is not something we can fully automate (at least not yet… I’m looking at you Elon) and thinking that we can will surely decrease the quality of our products. The work of a good tester cannot be boiled down to a few discrete and repeatable steps that a machine can reproduce, the tester is doing so much more.
With all that being said I feel compelled to clarify, once again, that automation is a powerful tool and should be used by any serious tester whenever the situation calls for it. But only then.
