Automated vs Manual Testing

Recently, there are some friends from Hong Kong started asking me about the proper test strategy for their teams/ companies. As part of the discussion, automated versus manual testing being brought up. Development cannot keep up on just doing manual testing, what should we do? What is the automation percentage we should target? What is the requirement for automation testing?
I am going to put together my thoughts into a series of posts on these topics, from this “Automated vs Manual Testing” to how to convert from manual to automation.
Assuming you have basic knowledge on manual testing and automated testing.
One paradox I have heard recently is that:
“Automation cost as much as manual testing, some times even more. What happens is that automation is harder to develop than anticipated, you need to maintain both the automation and the system moving forward too. (dotted line — You shouldn’t go automation)”
I have to admit that bringing up automation stack from scratch can be costly, that’s why there are continuous integration (CI) systems providing basic support to bootstrap the unit testing part. However, the critical problem is that this statement is not comparing the two testing approaches thoroughly.
Cost
Manual testing is not just about throwing a ball at the wall and hoping QA can manage and provide valuable bugs, this is also a common perception about blackbox testing. The truth is, manual testing still requires proper test planning to make sure the coverage, even though QA may not need to know how the feature under test is implemented, but they have to be aware of the user scenarios it supports. Keeping the test plan up-to-date is required for manual testing, since who is executing the test plan usually is different from who is changing the scenarios.
Besides, there are tooling needs to enable manual testing, batch user creation, maintain user(s) in certain state for testing, fake data for testing certain flow (e.g. fake credit card for payment flow), etc. Maintaining all kinds of test data is not trivial. Often this data has no automation guarding, which means engineers may break them without awareness.
Automation testing requires team to implement and maintain the tests onward, and sometimes when new requirement comes in, new tooling will be required. The piece that less frequently changed is the backbone of the testing pipeline, and the integration with continuous integration (CI) or continuous deployment (CD) system.
Isn’t the cost look similar between automated and manual testing? Someone may argue that automating tests is time consuming, fixing a bug seems more productive than writing a test. Usually there are two reasons that makes writing tests time consuming. First, the component has low testability; Second, tooling is not ready. Resolving second reason is simple, find the right tooling or build your own. For the first cause, one have to think about how to break down the component into smaller testable units. For instance, Uber introduced Riblets to tackle the lack of testability on giant controller class in MVC model.
If the cost is almost the same, why should one switch to automation if there is well established manual testing system?
Long-term Investment vs Short-term Reaction
Automated testing is for making sure sustainability of the development, once you have the testing pipeline up and running, you can start introducing automated tests, scale up to handle thousands of tests running in parallel, to shorten the turnaround time of test signals on each changes. However, this cannot be done instantly, everything has to be built piece by piece, along with the product evolves. Sometimes, introduce new test category can be time consuming.
Manual testing usually gives people an impression that it is more reactive, since the bring up cost of a new test resource (e.g. human) is low, as long as you have good documentation, tooling ready. However, it cannot really scale up with this, imagine the team hired hundreds of people doing manual testing, that’s costly and inefficient, not to mention managing this army is challenging too. Besides, how consistent can people be when performing tests?
There is no perfect solution, or one always better than others. We should understand the pros and cons, and apply one that fits the situation. In reality, most products use the mix of manual testing and automated testing. But along the growth of a product, the automated testing portion should grow at the same time. That’s my two cents.