Thoughts about Test Automation implementation.
I dare to say that we live in the gold age of test automation. We have so much content on how to get started, boiler-plates to give a base to your project, endless frameworks to avoid writing the basic stuff, AI claiming to make UI tests more assertive… the list goes on.
I’m pleased to see that, as a few years ago, we rely solely on web test automation (ex: selenium), and testing things that were not web pages was hard.
Combining my experience with what I see on forums, groups, Linkedin, and so on, I noticed that people started to get back to the basics. They felt lost on what they should test if there is a test pyramid or not, what documents they should write, and the real value of exploratory tests.
Either there are discussions about automation (on the generic implementation phase) or talk about if you were going to automate or not. Still, there is nothing about that middle section which is, which language to use, which framework makes the most sense, or which pattern is better for a particular set of problems.
It’s good all good, but can you understand the gap?
Because it’s easy for me to say, “Go use Cypress for UI tests. Here, use this tutorial to write your first test” or “Use BDD in automation to make the documentation come alive,” but it’s a whole new conversation when it comes to “Does Cypress makes sense for this UI test suite? Is my application compatible with Cypress? Should I use page objects? Who is responsible for writing the BDD sentences, and does it make sense to have all this work?”
I don’t know how much experience do you have in software development (test automation IS software development), and I bet that you at least heard about discussions on proof of concepts and benchmarks of specific languages to solve particular problems. Something like “we use Go language in this particular case because we need this service to be fast” or “we need something reliable and robust that only a JVM can provide.” Still, I never heard something similar when it comes to testing, and I’m in this business for more than a decade.
People use technology and tools without even asking why they are using them in the first place. This behavior is the classic “Maslow’s hammer,” and it’s easy to go for that path when you don’t have a solid base in software development. I’m not trying to be above anything as I feel in this trap for a bunch of years since I started to automate tests.
I’m also not saying that you should go through the “pain” of analyzing everything over and over again every time you start a new project. It’s ok to have a utility belt that you can count on, and the idea is that you need to be aware of the tradeoffs and be able to justify your decision.
Remember: They are utilities, not immutable truths.
Let’s deal with some daily life examples.
I’ve selected some situations that I see quite often in the testing community and wanted to share them so you can understand a bit more about what I’m trying to say.
Gherkin for API testing:
If you are familiar with the BDD process (the language is not the framework), you understand that you describe how the software should work from a user perspective. It encourages you to collaborate on writing scenarios as if the user was using, and I’m yet to find a user that behaves like this:
“Give I send an HTTP GET request,
When I get the response,
Then the headers have content-type as application/JSON
AND the status code should be 200”.
You can’t even sell this as documentation.
Can you image testing sophisticated API payloads using this pattern? It’s not just hard to read, but you also have the whole code as an abstraction that points to a method whose annotation is a REGEX, meaning that you are going to have a hard time trying to maintain its codebase.
There are way better tools to handle API documentation like the postman or a combo of tools that can manage those tests like pytest + request or RSpec + faraday, or jest + axios + unexpected.js, java + rest-assured, etc.…
Testing web components using selenium:
Have you ever seen a unit test run in your life? Is it usually fast, right? A suite with a dozen tests can run in less than a minute. Now check just one scenario of your UI test suite that asserts if the form has boundaries check implemented on each input.
If you are using a ruby + capybara +rspec combo, the chances are that firing up the browser with a specific context might take at least 10 seconds, not to mention all the procedures to reach that particular form. A simple check might take a minute or two.
I’m not even counting on the environment availability, network failures, apps not available because of deployment processes, you name it.
Now, did you know that most of the modern web development frameworks implement some test client or a way to start a specific component (react and vuejs), making testing those types of stuff really fast and somehow easier? You can take a look at the react test utils, Cypress, or how jest tests the components from the framework your project uses without actually firing up a browser.
Using Cypress for end-to-end tests:
Have you ever questioned why the creators of Cypress say that using page objects is not recommended with their framework?
Do they want to impose new technology? Maybe, but the main reason is that if you check on how they implemented their test runner and the way Cypress interacts with the application, it is clear that it's meant to test react and vuejs applications, usually components or single-page apps, but then you see people doing extensive end-to-end tests, creating huge setups, making monsters instead of cypress scripts, for the sake of the hype.
And if you ask them why they are using Cypress, I bet that they can’t explain.
Of course, you can enter the hype train, but beware of the commitment you make with the technology you choose. Take into consideration the community around, the company, or the group of people that maintain it, and difficult it would be to migrate your codebase to another framework.
Testers should use the same language as the developers:
This saying is tricky because it involves a bunch of variables such as the seniority of the dev team and the tester and the type of test that they are doing.
A senior dev team is not defined by the languages they use but by the problems they solve. I saw a dev team switch from ruby to typescript on new projects because graphql tools are native to the JS world. Have you ever tried to use the official graphql gem for ruby? Its madness. The same goes for the python library of Kafka.
So, if a senior dev team can understand the tradeoffs of libs and languages, why the testers can´t do the same? I mean... Why have graphql tests with ruby when you can do the same thing way faster with less dependency with typescript and a graphql client?
This is just an example that I was involved in and did the right call switching to javascript vanilla ES6 and then moving to typescript.
I know that you may be asking why a QA needs to think about all of this when the majority of the cases in their professional life will be testing web apps or mobile apps.
The thing is that even when you talk about web and mobile apps, you have different approaches, different tools in different languages that have their pros and cons. As developers of tests, you need to incorporate the development process into your routine to make the most productive decisions.
Have a beautiful day :)
Send some love to Ramses Sacol that was kind enough to review this and help me do a better rant :)
