Increase Test Coverage: Automation Test Analyzer

Photo on Unsplash

We as a search team at Trendyol, our mission is providing the fastest, the most relevant and personalized experience to our customers. At this point, we develop variety of new features in every sprint.

In this case, we have to make sure that our existing features are stable after the deployment of new features. High percentage of test coverage is fundamental for our services like all development teams. However, it is not the only difficult to keep test coverage high also it is really challenging to measure and make sure of it. Since it is crucial, how can a team measure a test coverage, and be sure ?

In this article, it will be told that how automation test analyzer increase the test coverage, how it measures the coverage base on customer behaviour and implementation to test automation project.

Automation Test Analyzer

“Automation Test Analyzer is a utility project that we can use to determine the strength of our test automation by comparing automation parameters with actual request parameters.”

Overview of Automation Test Analyzer

According to the overview above, the project consists of 3 main parts. It takes all required inputs in the first part, then it will calculate the missing scenarios. At last, it will generate the missing scenarios in the last part.

Part 1

It gets the requests from latest job of test automation pipeline. At the same time, it takes customer requests from the data source that hold the customers input data. Also, it eventually reads the necessary configs from environment variables. These variables are;

• IGNORED_PARAMS: It ignores the parameters that have no effect for test cases, e.g, parameters which are store information, restaurant addresses, are not necessary because they do not have any effect on business cases
• Actual Data Source: Database where we keep production client requests
• Data Source Info: Required information of the connection

Part 2

In this section, it process the inputs that were taken from the first part. These processes are respectively;

• Creating the automation parameter request list
• Preparing the client request list
• Cleaning the duplicate request from both list
• Sorting both list in alphabetically
• Calculating the coverage with automation test request and client request
• Generating the final list that includes the missing scenarios

Part 3

It generates the test scenarios with the missing request parameters that are generated in the second part.

• Taking the missing parameters as input and generating the test scenarios in Gherkin format

Let’s have a look at more detail

Client Requests

Automation Requests

Let’s think about there are two lists. One of them is for client requests, and other one is automation requests.

In this case, parameterA and parameterB seem to be ineffective for business, so they can be ignored by IGONORED_PARAMS.

IGNORED_PARAMS: parameterA, parameterB

Client Request Parameters

Automation Request Parameters

From now on, there are two parameters lists. However, they have duplicated values and non-sorted. Therefore, sorting and removing duplicates are necessary processes.

Final Client Parameter List

Final Automation Parameter List

As it is clearly seem that there are two parameters lists that have unique and sorted value. Hence, merging the lists is possible for observing the missing scenarios.

Final List

In this step, there is a final list that has the missing parameters, so test cases will be written with these missing parameters by Automation Test Analyzer. For instance;

Scenario: Case #
When I am  <parameterA>
And I write <value> in "parameterC"
And I write <value> in "parameterE"
And I perform get request to "Endpoint Name" endpoint
Then I should see response is 200

Finally, missing test cases are ready to implement for automation project.

Benefits

• Increase Coverage
• Simulate Customer Request
• Decrease Cycle Time
• Less Workload
• Stay Up To Date

Automation Test Analyzer is for decreasing the test effort while writing the automation test cases. Thus, it decreases the cycle time of a team. Moreover, it helps us to stay up to date on test automation project because of the reflection of customer behaviour, and it increases the test coverage and keeps it.

Results And Struggles

• 1566 New Test Scenarios Generated With Analyzer
• 1869 Total Test Scenarios
• Increased Test Coverage
• Test Coverage Percentage
• Test Data

According to our experiences, it increased the number of automation test scenarios from 303 to 1869. The challenging part of this project was creating test data. It led us to succeed increasing automation test coverage with 1869 test scenarios based on parameters.

Conclusion

At Trendyol, getting rid of manuality is one of our main goals. Automation Test Analyzer is an example of this goal. Improving and calculating the test coverage and implementation to automation project via Automation Test Analyzer are mentioned in this article. In the future, we aim that project will stream and generate test cases with relevant datas. Also, it will create a merge request for test automation project.

Thank you for reading and feel free to contact us if you have any questions.

Oğuzhan Erdem
Yusuf Kemal Özcan