Testing Automation — Making It Right #1: Theory
This is the first part of the series.
#2 — Test Types and Integration Tests
#3 — Test Machine Learning Based Applications
About the series
Over the years I have worked on lots of projects — as a developer, as a manager and as an entrepreneur, from small and medium startups to large enterprises, across the fields of machine learning, big data and web applications. Its a good time to reflect and share tips and insights I have learned in creating effective testing automation. I would like to discuss testing from a methodology point of view, and debunk some misconceptions.
I never had QA engineers in my greenfield projects. Nevertheless, these projects’ were much higher quality than projects I was involved in that have QA engineers.
Not only I feel that the projects without QA engineers have less bugs, regressions and crashes — I much more enjoyed developing these projects. The development was fast and effective, and I could concentate in the code.
By “projects without QA engineers” I don’t mean there was no testing. In these project, there were two rules for testing: (1) All tests must be automated, and (2) all tests must be written by the developers, and not by dedicated QA engineers.
Let’s first start with setting goals for testing automation. What exactly are we trying to acheive by having automation tests?
Setting goals
Engineers usually find automation beneficial because it increases quality, reduces regression testing time and allows you to deliver fast.
I agree, but I would like to tackle it from a different angle. What I am about to say might seem obvious, but I would like to shift the argument from end results to how we can improve our development processes and adopt more effective development methodologies.
So, with automation tests…
- The feedback to the developer on the correctness of her code comes much faster. This is one of the key enablers for fast development.
- Dependencies between teams are dramatically reduced. Not only that you don’t need to coordinate between the dev team and the QA team; With automation tests you’re capable of developing the server independently from the UI; You don’t need to wait for other team’s microservice to be ready; You don’t need other teams to analyze a bug or a support case, before it comes to your desk. You can easily add a new test that reproduces a field case using the same input + output.
- Customers won’t get the same bug twice. Sounds trivial, ha? tell that to enterprise customers.
- New or inexperienced developers are more limited in the damage they can do to your production environment. Not only it will be hard for them to ruin the production, when I had a [very] weak developer, I just made a massive code review on his tests. Shitty code, but works.
- Development considerations are written explicitly. That’s the best way to document things. Developers cannot ignore them. They will see immediately why their code contradict the ideas behind the base code.
- You have a clear gateway before production deployment. Think of a big refactor you did. How much were you afraid before deploying the code to production?
When these are your goals, along with the common ones, your whole methodology may change. It’s not just that you’ll add few QA automation engineers that write tests on top of the existing flow. The goals above put the responsibility mainly on the developers:
If a QA engineer will write the tests, the feedback loop to code quality is longer; The ability to write code independently from other developers / teams doesn’t exist; QA engineers are not familiar with code considerations, and therefore they cannot leverage the testing as a documentation tool.
Misconceptions
When I am speaking about the need of developers to write testing automation, for some organization its trivial (OK Google). But for some, I get strong resistance. The main two claims are:
- The development become much slower.
- Its frustrating developers.
I am not going to argue about #1, because if you are writing it correctly — the development becomes faster. Not slower. Those who claim #1 didn’t work on a project where the testing automation was effective. The reason for the fast development is simple: The main time consumer is bureaucracy and dependencies between teams. Testing reduces both. There are more reasons why testing makes the development faster, but as I said — I’m not going to convince you why its true. Before claiming the development is slower, you should work in a place where there are good testing.
Regarding #2 — “Its frustrating developers” — its true. It is frustrating developers, but mainly the poor ones. What I can say is that in my experience, the dependencies between teams, the inability to know in advance whether the code works, deploying patches in the middle of the night — all of these, are much more frustrating. Therefore, I prefer writing tests.
Do we need manual QA engineers?
This question must be asked: Do we need manual QA engineers? Should the developers do everything?
In my opinion — in most cases we don’t need them. They usually become bottlenecks, and it prevents the project to develop in the right way. Having manual QA engineers reduces responsibility from the developers, the quality feedback to the developers arrives a while after the code has been completed, and… its very expensive. More engineers, more people that has to be involved in code & bugs, more bureaucracy…
BUT,
There are few cases where I would take manual QA engineers (in addition to automation tests written by devs):
- On-prem products: fixing on-prem product problems costs the organization a lot more than in SaaS. Coordinating with customers, getting the logs, inability to control the environment, deploy debug patches, etc. Therefore, we want QA engineers to be “safety net” before going GA.
- You may want to have QA engineer to independently test the customer experience, or test the global system experience as a whole. Its not a must, but it can bring value. You need very few QAs for that, and they shouldn’t be a gateway before deploying the code in production.
Next: Test types
