Manual Test Cases vs. Automated Test Cases — who wins? Round 1.
I am for a decade already in Software Testing, but a few months ago I reread a book “Software Testing. Basic course” (basic course!). This book is available only in russian, it is made by EPAM training center and unifies experience of EPAM engineers in software quality process. I’ve read it to review and sync my understanding of the basics in a discipline that I have a lot experience in.
In a book I found several interesting things and they are so important, so I decided to share them with you in several blog posts. One blog post for one basic yet important thing.
This post is about 1 crucial understanding in “Manual vs. Automated test cases” topic. I hope you’ll like it.
Manual tests vs. Automated tests curve
This is one of the best understandings for automated test cases that I ever met:
Automated test cases — are “cement” for your product functionality
This is muchos importante, amigos! :) Indeed: there’s often no need to cover functionality with auto tests because this functionality is not yet completely defined in its core, it changes, it transforms. You’ll boil the ocean trying to cement what does not want to be stable yet!
Functionality “has not yet decided” in what form it “wants” to live in this world. It is literally burned. It looks for its way, a way of realization, way of implementation in respect of all the limitations of the material world. An idea (non-material thing) standing behind the function goes through implementor (a human, a developer) and appears in a real world (becomes a material thing, a button on UI, for example). Don’t solidify, do not freeze an idea fast. Let it get the form first. And it takes time. How much time? Let’s find out!
This understanding is perfectly displayed in “Software Testing. Basic course” book, where an author (Svyatoslav Kulikov) explains strengths and weaknesses of manual and automated testing (I redrawn his images):
Manual test cases are usually much faster to be designed, than automated test cases. This means, that using manual tests you can start testing much earlier, than using auto tests. In some cases this benefit can be winning.
Also, Svyatoslav Kulikov introduces a fast-average ROI formula for automated test cases:
and according to it, if we have pretty reasonable values for the projects:
then we’ll get the following graph:
I.e. we will find out that 12 test runs will show us a border of efficiency of automation appliance in this case.
To display this concept and to union it with manual test cases advantages, I’ve made another picture:
— i.e. for the same conditions of time efforts manual test cases could be developed and executed N times (for example, the same 12 times)— and you’ll get the picture for your project, when automated testing starts to win your manual testing to define that N for your company.
Conclusion and Takeaways
We can see from practice and charts above, that in some cases tortoise wins by a hare. That’s why don’t blame manual testing as “dead” or manual test cases as “not needed at all”. There are a lot situations when manual test cases are much faster, much cheaper and do their job better, than automated test cases.
Automate test cases for the feature when you sure, that you need to “cement” that functionality. I.e. when you’ll run that tests for at least N* times during their lifetime, before they get obsoleted. Calculate this N* for your company, for your team, for your projects.
N* times — N depends on your project time efforts and could vary from 1 to several tens. But a reasonable value, displayed in the book and in article, is 12.
In our company we use both manual and automated testing. We use all the benefits of manual testing: we throw it to the front line to gain confidence about software quality as much as we can. And when we’ve got confidence (the bugs found, fixed and verified or no bugs found), then we can think about to “cement” some of them with autotests.
Further reading
P.S.: I hope you liked an article! Please, don’t hesitate to give it some claps — so more people could read it!
Thank you!
