The Importance of A/B and Exploratory Testing (ET) in Trendyol iOS App
“The biggest risks must be dealt with first.”
In this article, I will talk about the testing approaches in the Trendyol iOS team.
How is the testing process progressing in a sprint?
Why should test data be prepared?
What is A / B Testing?
Why is exploratory testing done?
How should the test scenario be prepared?
How is the testing process progressing in a sprint?
We are progressing in Agile structure by running 1-week sprints in Trendyol iOS team. With grooming, we also get an idea about the work to be taken in the sprint before planning.
While trying to determine the test effort by examining the work through the use cases prepared by the Product Managers and the Zeplin designs prepared by the UI / UX designers, on the other hand, we explain the test data that may be required for the business and what we expect from the backend services.
Why is it important to pre-determine test data?
In grooming, we start to prepare test scenarios based on the use cases we have obtained from product managers, and we determine the areas where test data may be needed.
It is very important for us to prepare these data before the work comes to the test phase.The reason for this is to ensure that the developer can use these data while developing the business. And likewise, to be able to continue the process quickly when it comes to testing. Recognizing the data need late will affect the work effort and cause the sprint to be perpetrator.
We mostly prepare these data for production environment as well as test environment. Before the work goes live, we definitely realize business scenarios related to production data.
Process of Works with A / B Test
Since we are a mobile team, A / B testing is indispensable for us. we test a feature with A / B at almost every sprint :) The A / B test is a method of comparing two different versions of the application with each other, if I describe it briefly. We also observe and measure the behavior of users who experience these 2 different versions through events.
How do we start the testing process in A / B task?
In A / B feature, configs and events are determined to measure the old and new versions, namely the A and B version. First of all, we verify that pull request configs opened when it comes to the test are added to the test and production environments. If there is a config that is not added, we ask the developer to update it.
How do we analyze work to minimize test effort?
Since the test effort of A / B task may have phases such as design changes, business changes, and the addition of events, it is usually too much. The easiest way to minimize this test effort is actually analyzing the work well.
We usually start doing this analysis with exploratory testing.
What is Exploratory Test?
When we look at mobile application tests, one of the biggest challenges is the variety of devices and operating systems. We experience this diversity quite a lot while testing feature with A / B.
If I talk about an example that I lived in the sprint;
* A / B task had come to the test.
* I builded a package on any iOS device and started testing.
* I was opening UI bugs by comparing with the designs on the Zeplin.
* Then I started to try business logics and suddenly I had a crash.
* I wonder if it only happens on this device? Is it version based? questions like it started to spin in my head.
* When I repeat the same scenario with a different device and version, I saw that there was no crash.
This event that I was experiencing actually made me realize that. “It didn’t make sense to go into business to open a bug as soon as possible.” Because rather than finding bugs, it was my primary task to identify the devices and versions that should be tested.
While trying to find cosmetic bugs in iOS 13 version, I missed that the expected function for iOS 12 did not work and caused a crash and I could not prioritize it in sprint. Since the crash was received and noticed late, the testing process stopped until a solution was found.
Such problems are very likely to occur. So we decided to do exploratory testing in order not to experience this problem again.
Instead of trying to open a bug,
- Does the software perform the most basic and most important functions that the software needs to do in different versions and devices ?
- Does it perform the most basic and most important functions, does it display the most viewed screens as it should ?
- Does crashes or critical problems occured in the most used sections we check ?
If there is no problem, we start doing design checks and opening bug records.
How should the test scenario be prepared?
Writing test cases in steps is very important for readability. Because everyone who reads the script should be able to make the same meaning. It should be stated which prerequisites must be fulfilled. Existing flow and expected results should be written.
We follow our test scenarios within the team on the kanban board we call “QA Automations” on JIRA.
What is the importance of writing different test scenarios for the A version in task with A / B?
I wanted to share as an example a test scenario and flow of the "Warning when more than one discount coupon or code is used in the basket" logic in version A - that is, existing in the store flow - and the ongoing logic. You can see the flow of the steps in the test scenarios on gif.
For the A version of the A / B task, we must always make sure that the scenarios in the latest Trendyol version in the app store work correctly. For this reason, while preparing our test scenarios, we also consider the ongoing scenarios in the A version of the A / B task.
What is the importance of writing different test scenarios for the B version in task with A / B?
In version B -the new business- I wanted to share a test scenario and flow of the “Information popup view in Trendyol Wallet Coupon” logic as an example.
You can see the flow of the steps in the test scenarios on gif.
Task with A / B, as we mentioned, also include new business logics. Therefore, we must make sure that the scenarios for these newly added businesses are working correctly.
For this reason, while preparing our test scenarios, we are also considering the newly added scenarios in the B version of the A / B task.
