Manual Testing to Automation Mastery in QA
How we managed to create an automation test suite that significantly enhanced our QA process and boosted efficiency tenfold, overcoming initial challenges and delivering robust, reliable testing solutions.
I joined QA because I Googled it during college placement season and found a link claiming it was the happiest job.
Turns out, it was a sponsored ad.
I applied for a QA job, not meant for computer science students, mainly to dodge the competition. Plus, the pay was double what SDE roles at mass recruitment companies in India offered.
Turns out, money does talk — and it told me to test software!
The job turned out to be pretty interesting, especially with the random bugs like API responses mysteriously setting to null, chipping away at the developer’s confidence, one bug at a time.
Life improved as I discovered new tools and created my own framework to fast-track manual testing.
I got better at finding all sorts of bugs, not just the null errors. My documentation speed more than doubled, and testing tasks that used to take all day were now done in less than an hour.
This sounds like a great improvement but, what if I told you I was able to 100x both my documenting and testing using automation!
One fine day I was asked to build software that is twice the size of the actual software to test the actual software!
The internet was void of code to test APIs. My testing doesn’t stop at checking responses.
It did not stop at checking the database.
Since the product I was working with is an integration-type product (it gets values from the user and sends the same values in different formats to other applications)
For example: the client provides the order details to our shipping label (label that you see on your Amazon packages) generating software. The values are restructured to other formats so that logistic aggregators (companies that associate with multiple courier and shipping companies to provide e-commerce logistics solutions) can process the order and assign courier partners to deliver the package.
How do I test that?
I even found that the software needs to be designed in some way so that it should be testable or should be easier to automate.
Version 1 (Noob-ocalypse: The Ultimate Beginner’s Battle):
With little knowledge and taking reference from an already stable test suite, an MVP of the test suite was created. There wasn’t even a proper plan for automation test cases.
Tools and Frameworks used:
- Junit4 (framework for writing testcases)
- RestAssure (framework for API call)
Features it had
- The application made POST calls to API and the response was validated
- Assert statements(checks in if the expected value and the actual value are the same) were added. Works well for tax calculation calls
What was it lacking?
- DB checks — We did not know whether the values stored in the database were the same as what was sent in the payload
- There are calls made to third-party applications. We need to check if whatever the values sent are the same as what is sent by the client
Version 2 (Noob-ocalypse Reloaded: Database Disaster)
There was already a Testsuite which was built and maintained by seniors of my company. The application was using a dump file which was getting restored every time before each test case.
What a waste of time 🕰️ (But…I regretted this decision to the core).
I was too confident enough that I could create a better architecture that would delete only the required added values in the DB to save time in loading the dump.
Tools and Frameworks used:
- java.sql package which provided APIs to execute database queries and retrieve query results for asserting
Useless feature it had:
- Manually created a dump for constant values like default GST percentages, country, zip codes, etc, and ran it before running the automation test suite.
- There were functions that only checked the corresponding tables whether there was data present but not the value of the data. For example: assertNotNull(result.next()) 🤦🏻♂️
What was it lacking?
- The suite was able to test the software but when one test failed, pretty much every case following it failed because when the case failed the delete queries did not run leading to failure of other cases 😩.
- The request and response data should be the same as what is in the database. But that was missing in this version of the test suite.
- There were 3rd party application calls made. Those calls need to be mocked and find ways to cross-check the values being sent to those applications.
Version 3 (Zen-ocalypse PDF checks and Log checks)
The robustness began to take shape after this version. With the given coordinates along with a library called PDFBox, we were able to verify whatever that was provided in the database was printed properly in the invoice PDF. The Mockoon app was used to mock API calls made to logistic aggregators. The request and response logged to ELK (Elastisearch- a place where information about all API and their response are stored for debugging purposes) was retrieved.
Tools and framework used
- PDFBox is used to values from a particular coordinate. This helped in cross-checking the values printed in the invoice with the values sent to generate the invoice.
- Rectangle2D to create a red box to represent the area the PDF box is converting the portion of PDF to text.
Features:
- Check the values in an invoice PDF file
- Checks on the same values sent to 3rd party were a game-changing feature of the test suite
Improvements needed
- The Test Suite seemed to be functional but quite far from complete with the absence of database validations.
- Remember, when one test failed, it often resulted in subsequent tests also failing due to the unexecuted delete queries. It could have been easily addressed with @AfterMethod(testng) or @Before(junit4) annotations.
Version 4 (Dump and Better Dump-ocalypse)
In this version, we created separate dump (just a copy of the whole database in a file) files for each major scenario and restored the dump every time a test case was run.
Each case will be executed on the same database with the same values as the previous case, resolving the major issue of dependent test cases failing caused by unexecuted delete queries.
But there was an elephant sized performance issue in the test suite. Each case took ~1 min to run, by this time the suite had ~400 cases which resulted in ~6.5hrs 🤯 of running time.
A Piece of Code to the Rescue
There was some odd piece of code in the test suite built by our seniors called the “db_connect” which was a micro-service running in the machine that was running the application.
This micro-service received some instructions from the test suite running and based on the instructions it restored dump files hosted in a GCP Bucket (A Google-owned place for storing files) in the database after each test-case.
The dump restoration was very fast as the network GCP instances were in the same network which reduced the restoration from 1 min to just 4 freaking seconds!
Version 5 (Reporting and Automating the Automation-ocalypse)
Here was the birth of true automation took shape with some help from a paid testing tool for keeping record of automation cases, generating test reports before each release of software.
We did choose a software that was quite pricy costing 35 dollars per month per person. We are sharing 5 accounts among 10 QA engineers since sharing is caring right 😉.
We have also managed to include the testing process in Jenkins, so that we have automated not only the testing but also the deployment of the application in the test environment.
This saves more time for a quick TT match with colleagues while my electronic companion does my job.
Well, this is my automation journey. But we are not done yet, we still need to figure out how to get the code coverage of the test suite. New requirements pop up and I am shifting left (taking part in the solution phase, which is the phase before the actual coding takes place) and looking forward to learning more and sharing my experiences with you all.
Happy Testing!
Author: M M Kishore
Stay connected and stay informed with ProductPanda! If you’re a recent graduate who’s hungry for tech-knowledge, make sure to follow us 🎉 on our journey 🚀 .
