A CRISPY RECIPE FOR THE TESTING AUTOMATION
Usually, I don’t write recipes down. But this one was inspired by little kitchen chat with Andrii Maistryshyn, Head of SDR. One his question just led to another, and we sort of went carried away. Well,
Any application can be tested manually, while automated testing is worthwhile to be implemented for stable and long-term projects. Test automation enables one to decrease the testing time from days to hours. True story.
A hybrid of both approaches is the optimum way to get the best results in time.
This is what our client, Frontier Car Group, decided to do.
Automated testing has sped up the testing process (surprise, surprise). After we’ve wow’d FCG with the results on a small patch of an automation framework, they relied on us to cover most of the code with auto-tests.
Before even starting, this kept us awake:
Running a massive amount of tests in a short period — needs an efficient parallel execution
Pain-free passing complex business flows in tests pre-setup
Dealing with red pipelines — flaky false negative tests
To-walk-and-chew at the same time testing approach
FCG operates in more than ten markets, and the system has acquired different configurations for each. As for releases, they happen every single day with no room for mistake. The tests had to be run frequently, simultaneously, and effectively. Mocha Parallel, Selenium Grid, Kubernetes, and the Google Cloud Platform came in handy.
We run 3000 tests on one pipeline and up to 20 pipelines every day
Here are the numbers — We run 3000 tests on one pipeline, meaning its 300 tests scaled for ten countries. The first secret is that we don’t write ten tests, but instead write only one, flexible to different country’s configuration. Another secret — we’ve organized effective parallel remote execut ion. Using Kubernetes and Selenium Grid, we run all countries in separate jobs. One job consists of 50 simultaneously run processes. So, one pipeline with deployment takes us less than 20 minutes.
Requests. We need more requests. Bunch of them to form test objects. Here comes a Godsend tool — ApiBuilder. Actually, it’s the creation of ours. The app has complex business flows to be tested and a large amount of data to generate across different microservices. Also, each country has its own validation rules on those. That means a lot of time to set up a test object to initiate testing of a particular case. ApiBuilder is a project-specific tool. It allows one to quickly set up test objects, thereby decreasing feature investigating time and speeding up implementation.
For instance, we need to test if a bid could be successfully placed on a car at an auction. So the test case is a bid. The given is a correctly set up car, to move forward with testing steps.
All is needed now is to pass specific parameters. Else, the builder will use default ones. Pure magic. The validations and flows passed with four lines of code, and we had to write dozens before!
Tests were dropped on pipelines. That, of course, slowed down work and prevented us from moving on. We started to dig and have found real gold.
The pain in a neck was that locally passed tests failed on pipelines. That made it almost impossible to clear out the initial problem.
Seed random – another add-on we’ve designed. The seed is generated at the beginning of the test. It is used whenever any randomizer function called. So, all data used in the test grows from that seed. By using it locally, we play out the exact conditions under which the test has failed. It helps us to investigate a crash point.
So, we brewed the magic potion, which makes it possible to handle 3-day long QA testing tasks in a few hours. Here are the ingredients: Efficient Parallelization, API Builder, Seed Random, and sharpest minds!
And we continue our cooperation with FCG. Now, as a big QA team, we have new challenges, as well as new ideas — so no doubt,
To be continued
